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PREFACE 


This book was the basis of a single graduate course on the general subject of "performance" 
of communication networks for students from a broad set of backgrounds in electrical 
engineering, computer science, or computer engineering. The student was assumed to have 
basic familiarity with networking concepts as discussed in introductory texts on the subject, 
e.g., [139, 172, 220]. Also the student was assumed to have undergraduate courses in 
probability theory and linear (matrix) algebra. 

Background material on probability and statistics is reviewed in Chapter 1. Graduate 
courses on probability and stochastic processes in electrical and computer engineering tend 
to focus on wide-sense stationary processes, typically in order to study the effects of noise 
in communication and control systems. In two successive chapters this book covers Markov 
chains and introduces the topic of queueing. Though the continuous-time context is stressed 
(to facilitate the queueing material), the discrete-time context is covered at the end of each 
chapter. 

The remaining chapters pertain more directly to networking. Chapter 4 is on the subject 
of traffic shaping and multiplexing using a localized bandwidth resource. The next chapter 
describes queueing networks with static routing in the rather classical contexts of loss 
networks and open Jackson networks. Chapter 6 is on dynamic routing and routing with 
incentives including a game-theoretic model. The final chapter is a discussion of peer-to- 
peer networking systems, specifically those for the purposes of file sharing. 

In general, problems at the end of each chapter review the described concepts and cover 
more specialized related material that may be of interest to the networking researcher. 
Worked solutions or references for certain problems are given in an appendix. 

The length of the book allows time for about two weeks of lectures on material of specific 
interest to the instructor. The amount of instructor discretionary time can be increased by, 
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for example, omitting coverage of the concluding sections of Chapters 4, 6, and 7 that are 
largely drawn from the author’s own publications. 

I thank my wife Diane, the editing staff at Wiley, and several of my colleagues and 
students for their time spent reading draft manuscripts and for their sage though often 
conflicting advice. In particular, I thank my students Y. Jin and B. Mortazavi. All flaws in 
this book are the sole responsibility of the author. Please contact me to report errors found 
in the book as an errata sheet will be made available on the Web. 


George Kesidis 


State College, Pennsylvania 
kesidis @ gmail.com 
April 2007 


CHAPTER 1 





REVIEW OF ELEMENTARY PROBABILITY 
THEORY 


This book assumes some familiarity with elementary probability theory. Good introductory 
texts are [63, 192]. We will begin by briefly reviewing some of these concepts in order 
to introduce notation and for future reference. We will also introduce basic notions of 
statistics that are particularly useful. See [106] for additional discussion of related material 
in a computer systems context. 


1.1 SAMPLE SPACE, EVENTS, AND PROBABILITIES 


Consider a random experiment resulting in an outcome (or "sample") represented by w. For 
example, the experiment could be a pair of dice thrown onto a table and the outcome could 
be the exact orientation of the dice and their position on the table when they stop moving. 
The abstract space of all outcomes (called the sample space) is normally denoted by Q, i.e., 
wen, 

An event is merely a subset of Q. For example, in a dice throwing experiment, an event is 
"both dice land in a specific region of the table" or "the sum of the dots on the upward facing 
surfaces of the dice is 7." Clearly, many different individual outcomes w belong to these 
events. We say that an event A has occurred if the outcome w of the random experiment 
belongs to A, i.e., w € A, where A C 2. Now consider two events A and B. We therefore 
say that A and B occur if the outcome w € AN B. Also, we say that A or B occur if the 
outcome w € AUB. 
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Figure 1.1 A partition of Q. 


A probability measure P maps each event A C Q to a real number between zero and 
one inclusive, i.e., P(A) € [0,1]. A probability measure has certain properties such as 
P(Q) = Land 


P(A) = 1-—P(A), 


where A = {w € Q |w ¢ A} is the complement of A. Also, if the events {A;}’_, are 
disjoint (i.e., A4; A; = 9 for all i # j), then 


(Ua) = 5 P(A), 
i=1 i=1 


i.e., P is finitely additive. Formally, a probability measure is defined to be countably 
additive. Also, P(A) is defined only for events A C Q that belong to a o-field (sigma-field) 
or o-algebra of events. These details are beyond the scope of this book. 

The probability of an event A conditioned on (or "given that") another event B has 
occurred is 


P(AN B) 
P(B) 


where P(B) > 0 is assumed. Now suppose the events A, A2, .... A, form a partition of 
Q, i.e., 


P(A|B) = 


J Ai =Q and A;N A; =O foralli Æ j. 
i=l 
Assuming that P(A;) > 0 for all i, the Jaw of total probability states that, for any event 
B, 
P(B) = > P(BiA;)P(A)). (1.1) 


i=1 
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Note that the events A; N B form a partition of B C Q. See Figure 1.1, where B = 
(BM A4) U (BN A3) and BN A; =f fori = 1,2. 

A group of events Ay, A2. .... An are said to be mutually independent (or just "indepen- 
dent") if 


P (A a) =f] PAD 
ieL i€L 
forallsubsets T C {1,2,....n}. Note that if events A and Bare independent and P(B) > 0, 
then P(A|B) = P(A); therefore, knowledge that the event B has occurred has no bearing 
on the probability that the event A has occurred as well. 

In the following, a comma between events will represent an intersection symbol, for 
example, the probability that A and B occur is 


P(A,B) = P(ANB). 


1.2 RANDOM VARIABLES 


A random variable X is a real-valued function with domain 9. That is, for each outcome 
w, X(w) is a real number representing some feature of the outcome. For example, in a 
dice-throwing experiment, X (w) could be defined as the sum of the dots on the upward- 
facing surfaces of outcome w. Formally, random variables are defined to be measurable in 
the sense that the event X~'(B) = {w € Q | X(w) € B} is an event (a member of the 
a-field on Q) for all events B C R (belonging to the Borel o-field of subsets of R). In this 
way, the quantity P(X € B) is well defined for any set B that is of interest. Again, the 
details of this measurability condition are beyond the scope of this book. In the following, 
all functions are implicitly assumed to be measurable. 
The strict range of X is defined to be the smallest subset Rx of R such that 


P(X e€ Rx) = 1, 


where P(X € Rx) is short for P({w E Q | X(w) € Rx}). 

Note that a (Borel-measurable) function g of a random variable X, g(.X), is also a 
random variable. 

A group of random variables X1, Xo, ..., Xn are said to be mutually independent (or just 
"independent”) if, for any collection {By}, of subsets of R, the events {X; € B;}%_, are 
independent; see Section 1.9. 


1.3 CUMULATIVE DISTRIBUTION FUNCTIONS, EXPECTATION, AND 
MOMENT GENERATING FUNCTIONS 


The probability distribution of a random variable X connotes the information P(X € B) 
for all events B € R. We need only stipulate 


P(X <2) = P(X € (-x,72)}) 


4 REVIEW OF ELEMENTARY PROBABILITY THEORY 


for all x € R to completely specify the distribution of X ; see Equation (1.4). This leads us 
to define the cumulative distribution function (CDF) Fx of a random variable X as 


Fx (a) =P(X <2) (1.2) 


for x € R, where P(X < z) is, again, short for P({w € Q | X(w) < x}). Clearly, a CDF 
Fx takes values in [0, 1], is nondecreasing on R, Fx (x) — 1 as z — œ, and Fx (x) — 0 
as © > —00. 

The expectation of a random variable is simply its average (or "mean") value. We can 
define the expectation of a function g of a random variable X as 


ww) = f ~ iE: (13) 


where we have used a Stieltjes integral [133] that will be explained via explicit examples 
in the following. Note here that the expectation (when it exists) is simply a real number or 
too. Also note that expectation is a linear operation over random variables. That is, for 
any two random variables X and Y and any two real constants a and b, 


E(aX+bY) = aEX +bEY. 


E EXAMPLE 1.1 


Suppose g is an indicator function, i.e., for some event B C R 


g(X(w)) = 1{XWw)€ B} 
= T if X (w) € B, 
= 0 else. 
In this case, 
Ej) = PO eRe | dihts): (1.4) 
B 


where the notation refers to integration over the set B. 


The nth moment of X is E(X”) and the variance of X is 
o% = var(X)=E(X — EX)’, 


i.e., the variance is the second centered moment. The standard deviation of X is the square 
root of the variance, ox > 0. The moment generating function (MGF) of X is 


mx (8) = Bet; 


where @ is areal number. The moment generating function can also be used to completely 
describe the distribution of a random variable. 
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1.4 DISCRETELY DISTRIBUTED RANDOM VARIABLES 


For a discretely distributed (or just "discrete") random variable X, there is a set of countably 
many real numbers {a;}32, such that 


J > P(X = aj) =; 


Assuming P(X = a;) > 0 for all i, the countable set {a;}%, is the strict range of X, 
ie., Rx = {a;}%,. So, a discrete random variable has a piecewise constant CDF with a 
countable number of jump discontinuities occurring at the a,’s. That is, if the a; are defined 
so as to be an increasing sequence, F is constant on each open interval (a;,0;+1), F(x) = 0 
for x < aı, and 


Flai) = S— P(X =a;) = > plaj), 
j=l j=l 


where p is the probability mass function (PMF) of the discrete random variable X, i.e., 
plai) = P(X =aj). 


Note that we have dropped the subscript "X" on the PMF and CDF for notational conve- 
nience. Moreover, for any B C R and any real-valued function g over R, 


P(Xe€B) = SY pla) 
ajEB 


and 


xo 


EX) = S g(aj)pla;)= X gla)pla). 


j=l aERx 
To see the connection between this expression and (1.3), note that 


dF(x) = F'(x)dzr 


o0 


= X plai)6(a — a;) de, 


i=l 


where 6 is the Dirac delta function [164]. That is, ô is the unit impulse satisfying 6(t) = 0 


for all t Æ 0 and 
I dtd = 1. 


1.4.1 The Bernoulli distribution 


A random variable X that is Bernoulli distributed has strict range consisting of two elements, 
typically Rx = {0,1}. So, there is a real parameter q € (0,1) such that g = P(X = 1) = 
1— P(X = 0). Also, 


Eg(X) = (1—q)-9(0)+q@-g(1) 
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with EX = q in particular. 


1.4.2 The geometric distribution 


A random variable X that is geometrically distributed has a single parameter À > 0 and its 
strict range is the nonnegative integers, i.e., 


Rx = Z* = {0,1,2,...}. 
The parameter À satisfies 0 < A < 1. The CDF of X is piecewise constant with 
Fi) = 1-2"! 
for all i € Z+. The PMF of X is p(i) = (1 — A)X* fori € Z+. To compute EX, we rely 


on a little trick involving a derivative: 


EX = ip(i) 


IMs 


t 


= (1- OD Ait 
i=l 
difi 
= (1-A (È x) 
ai 


d 1 
1 
wa 


1-»X 


Similarly, the moment generating function is 


m(@) = 0-A) X (ea) 
1=0 
— 1A 
~ I= Àe? 


forefà < 1, i.e., @ < — log À. 


1.4.3 The binomial distribution 


A random variable Y is binomially distributed with parameters n and qif Ry = {0,1,...,n} 
and, for k € Ry, 


PY =k) = (f)ata ar 
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where n € Zt,0 <q < 1, and 


n _ n! i 
& ~ kin — ky oe 


Thatis, () is "n choose k," see Example 1.4. It is easy to see that, by the binomial theorem, 
Leh =K Se 


‘9 (ta-o = (¢+(1=@))” = 1. 


k=0 
Also, 


mo) = 2 (o — gyre 


E EXAMPLE 1.2 


If we are given n independent Bernoulli distributed random variables, X;, each having 
the same parameter q, then Y = 57", X; is binomially distributed with parameters 
nandq. That is, for k € {0.1,2,...,n}, the event {Y = k} can be written as a union 
on disjoint component events, where k of the X; equal 1 and n — k of the X; equal 0. 
Each such component event occurs with probability p*(1 — p)’~*. The number of 
such events, i.e., the number of ways the random vector (X1, X2,..., Xn ) has exactly 


k ones, is 
n! a n 
k(n- k)) kJ’ 


where n! is the number of permutations (ordered arrangements) of n different objects 
and the factors k! and (n — k)! in the denominator account for the k ones being 
indistinguishable and the n — k zeros being indistinguishable. 


1.4.4 The Poisson distribution 
A random variable X is Poisson distributed with parameter \ > 0 if Ry = Z* and the 
PMF is 
PNE O 
pli) = Se 
a! 


for? € Z7. We can check that EX = A as in the geometric case. The MGF is 


m(@) 


| 
iM 
oO 
p 
3 
È 
°, 
x 


exp((e? — 1)A). 


I 
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1.4.5 The discrete uniform distribution 


A discrete random variable X is uniformly distributed on a finite range 


Rx C R 


1 


for all x € Rx, where |Rx| is the size of (the number of elements in) Rx. Clearly, 
therefore, for any A C Rx, 


la 


P(X € A) Bx’ 


i.e., to compute this probability, one needs to count the number of elements in A and Rx. 


E EXAMPLE 1.3 


Suppose that a random experiment consists of tossing two different six-sided dice on 
the floor. Consider the events consisting of all outcomes having the same numbers 
(dı, d2) on the upturned faces of the dice. Note that there are 6 x 6 = 36 such events. 
Assume that the probability of each such event is at i.e., the dice are "fair." This 
implies that the random variables d; are independent and uniformly distributed on 
their state space {1, 2, 3, 4, 5, 6}. 

Suppose that we are interested in P(X € {7,11}), where the random variable 


X = d +də. 
That is, we are interested in the event 
(di, dz) € {(, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1), (5, 6), (6, 5)} 


with eight members. So, P(X € {7,1} = $. 


E EXAMPLE 1.4 


Suppose that five cards (a poker hand) are drawn, without replacement, from a stan- 
dard deck of 52 different playing cards. The random variable X enumerates each 
combination (not considering the order in which the individual cards were drawn) of 
poker hands beginning with 1 and ending with the total number of different poker 
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hands possible, (5? 


“*), where (7') is given in (1.5). That is, |Rx| = (=), Assume X 
is uniformly distributed (i.e., a fair hand is drawn) and suppose we wish to find the 
probability that a flush is drawn, i.e., P(X € flushes). As there are four suits each 


having 13 cards, the number of poker hands that are flushes is 4(2). So, 








4(23 
P(X € flushes) = a 
Similarly, the probability of a drawing a poker hand with a pair of aces from a fair 
deck is 





4 


where the term ($) = 6 is the number of ways to form a pair of aces and term (fẹ) is 
the number of ways to form the balance of the hand (i.e., 3 more cards) without using 


aces. 


E EXAMPLE 1.5 


Consider the context of the previous example but now suppose that we care about 
the order in which the cards were drawn, again without replacement. To compute 
the number of different hands, we count the number of ways we can permute 5 cards 
chosen from a deck of 52 without replacement. This quantity is 


52! 


52x 51 x 50 x 49x48 = 25l 


Note that a single combination of 5 different cards will have 5! = 120 different 
(ordered) permutations. 

On the other hand, if the cards are drawn with replacement (i.e., each card is 
restored to the deck after it is drawn) and we continue to care about the order in which 
the cards are drawn, the number of possible hands is simply 525. Note that in this 
case, a hand may consist of several copies of the same card. 


A group of example discrete distributions is given in Figure 1.2. 


1.5 CONTINUOUSLY DISTRIBUTED RANDOM VARIABLES 
The CDF of a continuously distributed (or just “continuous") random variable X has a 


piecewise-continuous and bounded derivative. The derivative f = F’ (ie, dF (z) = 
f(a) da) is known as the probability density function (PDF) of X. We clearly have 


F(z) = i f(z) dz. 
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Figure 1.2 Example discrete distributions. 


From this identity, we see that any PDF f is a nonnegative function satisfying 


[tee = 1. 


Moreover, 


E(x) = |  olz)f(x)de 
and, in particular, if g(X) = 1{X € B}, then this reduces to 
P(XEB) = f f(z) dz. 
B 


Finally, note that the range Rx = {x € R | f(x) > 0}. 


1.5.1 The continuous uniform distribution 
A random variable X is uniformly distributed over the interval [a, b] if its PDF is 


1 
no={ wu faxes, 
0 else, 
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where b > a. Clearly, EX = (b + a)/2. We can similarly define a uniformly distributed 
random variable X over any range Rx C R having finite total length, i.e., |Rx| < œ. 


1.5.2 The exponential distribution 
A random variable X is exponentially distributed with real parameter À > 0 if its PDF is 


Ae ** if > 0, 
f(z) = { 0 else. 
The CDF is F(z) = 1 —e~** for z > 0, EX = 1/X, and the MGF is 


A 


for 8 < A; see Section 2.1. 


1.5.3 The gamma distribution 


A random variable X is gamma distributed with positive real parameters À and r if its PDF 
is 


Arg? eT. ifr >o0, 
qe { 0 else, 


where the normalizing gamma function’, = te z™e~* dz. When r is a positive integer, 
we can integrate by parts to show that I, = (r — 1)!; in this case, the gamma distribution 
is sometimes called the Erlang distribution. Let u be the mean and g? be the variance 
associated with this distribution. We have the following identities: 


2 
u u 
AS and r= (1.7) 
Finally, the MGF is 
À r 
j= 8 
ma = (575) (1.8) 


for @ < À. We see that a gamma distributed random variable with parameters À and r = 1 
is just an exponentially distributed random variable. 


1.5.4 The Gaussian (or normal) distribution 


A Gaussian (or normally) distributed random variable X with mean u and variance g? has 


PDF 
fe) = ea exp (- E) 


ov2n 20? 
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Figure 1.3 Example continuous distributions. 


for x € R. The MGF is 


m(@) = expl u8 + $0°6"). (1.9) 


See Figure 1.3 for example continuous distributions. 


1.6 SOME USEFUL INEQUALITIES 


Clearly, if the event A; C Ap, then P(A1) < P(A2) = P(A1) + P(A2\A1). For any 
collection of events A1, Ag,..., An, Boole’s inequality holds: 


°(Ua) < 57 P(A). 


Note that when the A; are disjoint, equality holds simply by the additivity property of a 
probability measure P. 

If two random variables X and Y are such that P(X > Y) = 1 (ie., X > Y almost 
surely), then EX > EY. 
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Consider a random variable X with E|X| < oo anda real number x > 0. Since 
|X| > |X|1{|X| > z} > x1{|X| > x} almost surely, 
we arrive at Markov’s inequality: 
E|X| > Eri{|X|> z} = zE1{|X| > z} = zP(|X| > 2). 


An alternative explanation for continuously distributed random variables X is 


E|X| 


II 


I eteae > [atoe f zf(z)dz 


— Oo T 


IV 


[steers [epee = gP(|X|> 2). 


=O T 


Now take x = £?, where € > 0, and argue Markov’s inequality with (X — EX}? in place 
of |X| to get Chebyshev’s inequality 


var(X) = E[(X —EX)*] > P(X -EX|> €), 


P(X -EX| >e) < e7?var(X). (1.10) 


Noting that, forall @ > 0, {X > xz} = {e®* > e%} and arguing as for Markov’s inequality 
gives the Chernoff (or Cramér) inequality: 


Ee?X > P(X >r) 
=> P(X >z) < exp(—[x0 —logEe®*}) 
< — — OXI i 
< exp ( max([x8 log Ee l): (1.11) 


where we have simply sharpened the inequality by taking the maximum over the free 
parameter 0. Note the log-MGF of X in the Chernoff bound. 
The Cauchy-Schwarz inequality states that 


EIXY| < VEXAVEY?) 


for all random variables X and Y with the inequality strict whenever X Æ cY or Y = 0 
almost surely for some constant c. This inequality is an immediate consequence of the fact 
that 


2 
X Y 
(m n) oo 


whenever X # 0 and Y ¥ 0 almost surely. Also, note that if we take Y = 1 almost surely, 
the Cauchy-Schwarz simply states that the variance of a random variable X is not negative, 
i.e., that 


E(X?) - (EX}? > 0. 
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This is also an immediate consequence of Jensen’s inequality. A real-valued function g on 


R is said to be convex if 


g(pr+(1—p)y) < pg(x)+(1—p)g(y) 


(1.12) 


for any x,y € R and any real fraction p € [0,1]. If the inequality is reversed in this 
definition, the function g would be concave. For any convex function g and random variable 


X, we have Jensen’s inequality: 


g(EX) < E(g(X)). 


1.7 JOINT DISTRIBUTION FUNCTIONS 
For the case of two random variables X and Y the joint CDF is 
Fy y(a,y) =P(X <2,Y <y) forz,y ER. 


We can similarly define a joint CDF for more than two random variables. 


1.7.1 Joint PDF 


If X and Y are both continuously distributed, we can define their joint PDF as 


8 Fxy 
Ox Oy ` 





fxy = 


For any (Borel measurable) real-valued function g over R?, 


E9(X,Y) = a a Glas) fury (ede dy 


and, in particular, if 


axr) = 1} ( aes 


for some (Borel) A C R?, then this reduces to 


P (( - ) € a) = [| fxvenaedy 


(1.13) 
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1.7.2 Marginalizing a joint distribution 


Beginning with the joint CDF Fy y, we can obtain either marginal CDF Fy or Fy by 
simply taking limits: 


Fx(x) = jim, Fxyy(xz,y) and Fy(y)= Jim Fx y (x,y). 


To see why, note that 


Jim Fx y(z,y) = P(X < PY- < o0) 
= P({wEQ|X(w)<r} nQ) 
P(X <2), 


where we used the fact that, by definition of a (real-valued) random variable, the event 
Y < œ is the whole sample space Q. Similarly, one can recover either marginal PDF from 
the joint PDF: 


kas / keegi and fry) = Te fv (ey) ae. 


Marginal PMFs are similarly obtained by summation of a joint PMF. 


1.8 CONDITIONAL EXPECTATION 


Consider an event A such that P(A) > 0 and a random variable X. The conditional 
expected value of X given A, denoted j1(.X | A), is computed by simply using the conditional 
distribution of X given A: 


Fyja(z) = P(X <2|A), 


wixia) =f 2dF alo) 


In the case of a discretely distributed random variable X, simply 
CO 
w(X|A) = So a;P(X = a;lA), 
j=l 


where {a; }Ra = Rx. The conditional PMF of X given A is denoted px 4, i.e., 
px|a(a;) = P(X = aj|A) for all j. 

In a similar way we can define the conditional PDF of a continuously distributed random 
variable X given the event A, 


fxiale) = $ Fyle), 
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and thereby compute 


CO 


WXIA) =f afxiala) dz: (1.14) 


-o 


Consider now two discretely distributed random variables X and Y. We will define the 
conditional expectation of X given the random variable Y denoted E(X |Y ). The quantity 
E(X|Y) is a random variable itself. Indeed, suppose {b; }72, = Ry and, for all samples 


Wij € {w E Q| Y(w) = b;} = B; 
define 
E(X|Y)(wj) = u(X|B;), 


where the conditional expected value on the right-hand side can be denoted p(X|Y = 
bj). That is, the conditional expectation E(X|Y) maps all samples in the event B} to the 
conditional expected value u(.X|B,;). Therefore, the random variable E(X|Y) is almost 
surely a function of Y . 

Now consider two random variables X and Y which are continuously distributed with 
joint PDF fx.y. For fy (y) > 0, we can define the conditional density: 


fxy(2.y) 
gjy) = 
forall x € R. Note that fxjy (|y) is itself a PDF and, with this conditional density, the 
following conditional expected value can be computed 


pan s i af playa 


= 


where, unlike (1.14), the event {Y = y} has zero probability. Again, note that u(X |Y = y) 
is a function of y, say h(y), and that the conditional expectation E(X|Y) = A(Y). 

When X and Y are independent, the conditional distribution of X given Y is just 
the distribution of X and, therefore, E(X|Y) = EX. In other words, if X and Y are 
independent, then knowledge of Y (i.e., given Y ) does not affect the remaining uncertainty 
of the random variable X. 


E EXAMPLE 1.6 


For the purposes of a simple graphical example, suppose that the sample space 2 = 
[0,1] C R (but recall that, in general, Q can be a completely abstract space without 
ordering). In Figure 1.4, the previous case of jointly discrete random variables X, Y 
and E(X|Y ) are plotted as functions from Q to R. To further simplify the graph, we 
assume that these random variables are piecewise constant functions over Q and that 
the probability P is the (Lebesgue) measure corresponding to Euclidean length. 


'Bquivalently, E(X|¥) is o(Y )-measurable, where o(¥) is the smallest o-algebra over Q containing the events 
Bj. In this concrete way, o(Y ) quantifies the information content of Y. 


CONDITIONAL EXPECTATION 








Figure 1.4 Smoothing effect of conditional expectation. 


Figure 1.4 shows that E(X [Y ) is a smoothed (less “uncertain") version of X. Also 
E(X|Y) depends on Y only through the events 


B; = {Y =};}, 


i.e., the extent to which the random variable Y discriminates the samples w € 2. 
That is, consider a discrete random variable Z with range Rz = {c;}<, and define 
the events 


C; = {Z = cj} 


for all j. If the collections of events {B;}92, and {C;}72, are the same (allowing 
for differences of probability zero), then E(X|Y) = E(X|Z) almost surely. Note 
that Ry can be different from Rz, in which case 


E(X|Y) = A(Y) = E(X|Z) = g(Z) almost surely, 


with g Æ R. 


17 
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In general, E(X |Y ) is the function of Y which minimizes the mean-square error (MSE), 
E[(X — A(Y))?], 


among all (measurable) functions A. So, E(X|Y’) is the best approximation of X given Y. 
Again, E(X|Y) and X have the same expectation in particular, i.e., 


E(E(X|Y)) = EX. 


It is left as a simple exercise to check this property for the cases of jointly discrete or jointly 
continuous random variables X and Y considered above. 


1.9 INDEPENDENT RANDOM VARIABLES 


A collection of continuously distributed random variables X1. X2, ..., Xn are said to be 
mutually independent (or just "independent") if and only if their joint PDF is equal to the 
product of the marginal PDFs, i.e., 


for all real 71, 72, ..., £n. This definition is consistent with that at the end of Section 1.2. 
When X and Y are both discretely distributed, we can similarly define their joint PMF 
px,y and there is a similar condition for independence (the joint PMF is the product of 
the marginal PMFs). In general, a condition for independence is that the joint CDF is the 
product of the marginal CDFs. 
If the random variables {X; }"_, are independent, then 


ex = Tex. (1.15) 


In particular, for n = 2, this means that if X, and X2 are independent, then they are 
uncorrelated, i.e., their covariance equals zero: 


0 = cov(.X 1, X2) = E((Xı = EX: X2 = EX2)) = E(X)|X2) = EX EX3. 


The converse is, however, not true in general; see Problem 1.4 at the end of the chapter. 


1.9.1 Sums of independent random variables 


In this section we will consider sums of mutually independent continuous random variables. 
Our objective is to find the PDF of the sum given the PDF of the component random variables. 
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To this end, consider two independent random variables X; and X2 with PDFs fı and fo 
respectively; so, fx, x, = fı f2. Thus, the CDF of the sum is 


F(2z)=P(X +X% <2) = Te fe (11) fol) drgdx,. 
Exchanging the first integral on the right-hand side with a derivative with respect to z, the 
PDF of Xı + Xə is 


d 


f) =£ 


F(z) ys fi(ai) folz — 21) dx, forall z € R. 


Thus, f is the convolution of fı and fo which is denoted f = fi * fo. 
In this context, moment generating functions can be used to simplify calculations. Let 
the MGF of X; be 


m,(0) = Ee®*: =i fi(a)e®* dr. 


Note that m; is basically the (bilateral) Laplace transform [164] of f;. The MGF of X; +X» 
is 
m(0) = Ee®(*14%2) = Ee? X19°X2 — m (A)mo(8). (1.16) 


where the last equation holds because of the independence of X, and X2. So. convolution 
of PDFs corresponds to simple multiplication of MGFs (which, in turn, corresponds to 
addition of independent random variables). 


E EXAMPLE 1.7 


As an example, suppose X; and Xə are independent and both exponentially dis- 
tributed with parameter À. The PDF of X, + X2 is f, where f(z) = 0 for = < 0 and, 
for z > 0, 


f(z) = f filz) folz — 21) day = A?’ ze. 


The MGF of Xı + X2 is, by (1.6) and (1.16), 


m(0) = Ga). 


which is consistent with the PDF just computed. There is a one-to-one relationship 
between PDFs and MGFs of nonnegative random variables.? We can therefore use 
the MGF approach to find the PDF of the sum of n independent random variables 
{X; bre , each having an exponential distribution with parameter À. Indeed. the MGF 
of 57", X; is easily computed as 


m(@) = (44) E (1.17) 


?In this case, the MGF is a unilateral Laplace transform [164]: m(0) = [5° f(z)e®* dz. 
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This is the MGF of a gamma (Erlang) distributed random variable with parameters A 
and n € Z* and PDF 


rere r2 


falz) = ERD (1.18) 


for z > 0. 


E EXAMPLE 1.8 


Suppose that, for i = 1, 2, the random variable X; is Gaussian distributed with mean 
ji; and variance 07. Also suppose that X, and Xz are independent. By (1.9) and 
(1.16), the MGF of X; + Xo is 


m(6) 


exp(u10 + 40767) x exp(u20 + 4.0567) 
exp((m1 + u2)0 + 3(o7 + 03)07) , 


which we recognize as a Gaussian MGF. Thus, X; + X9 is Gaussian distributed with 
mean u + uz and variance o? + a3; see Problems 1.5 and 1.6. 


1.10 CONDITIONAL INDEPENDENCE 


If 
P(A|B,C) = P(A|B), (1.19) 


the events A and C are said to be independent given B. This is a natural extension of 
the unqualified notion of independent events, i.e., events A and C are (unconditionally) 
independent if 


P(A|C) = P(A). 


Note that (1.19) implies P(C | B, A) = P(C | B). 
Similarly, random variables X and Y are conditionally independent given Z if 





PXE€A(ZEB,YEC) = P(XEA|ZEB) 


forall A, B,C c R. Conditional independence does not imply (unqualified) independence, 
as we will see in the following chapter. 


1.11 A LAW OF LARGE NUMBERS 


In this section, we describe the basic connection between statistics and probability through 
the laws of large numbers (LLNs) [44, 62, 63]. Suppose we have an IID sequence of random 


A LAW OF LARGE NUMBERS 21 


variables X1. X2, X3,.... Also suppose that the common distribution has finite variance, 
i.e., 
o = var(X) 
= E(X —EX)* 
< œ, 


where X ~_X;. Finally, suppose that the mean exists and is finite, i.e., 
u = EX< œ. 
Define the sum 
Sn = Xi +X2 +- Xn 


forn > 1 and note that ES, = ny and var(S,,) = no?. The quantity Sn/n is called the 
empirical mean of X after n samples and is an unbiased estimate of p, i.e., 


Sn 
(=) 7 as 
n 
Also, because of the following weak LLN, Sn /n is said to be a weakly consistent estimator 


of u. 


Theorem 1.11.1. For all < > 0, 


—-pl>e} = 0. 
raze) 


lim P( a 


Proof: By Chebyshev’s inequality (1.10), 

















2 
p (|= ~ x >=) ee SE ae a 
n E ne 
Therefore, 
Sn 
lim p (|= - >e) = 0 
NOX n 
as desired. m 


The strong LLN asserts that, if E|X| < oo, then 


P (im Sn = n) z r 
n-x n 


In other words, S/n — p almost surely. So, S/n is said to be a strongly consistent 
estimator of u. A proof of the strong LLN, which implies the weak LLN, is given in 
[44, 62]. 
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1.12 FIRST-ORDER AUTOREGRESSIVE ESTIMATORS 


Given a series of samples X,, at "times" n € {0,1,2,...}, one may wish to iteratively 
estimate their mean. To this end, we can define X, as the "current" estimate of the mean. 
A typical problem in this context is that the underlying distribution of the samples is slowly 
changing with time. For example, the Transmission Control Protocol (TCP) estimates 
average packet round-trip times (RTTs) [139] (and "absolute" variation about the estimated 
average) to calibrate a time-out mechanism for acknowledgements of transmitted packets. 
Also, for packet queues in Internet routers, proposed active queue management (AQM) 
techniques such as [105, 165, 231] estimate packet backlogs and the number of long-term 
active TCP sessions. 

Since the distribution of X,, is changing, one could want to more significantly weight 
the recent samples X% (i.e., k < n and k ~ n) in the computation of X,,. For example, 
one might use a moving average (MA) of order 2, i.e., for n > 2, 


Xn = BoXn F BiXn-1 $ B2Xn-2, 


where 0 < (3; for all ¿ and $7, 6; = 1 (e.g., all 6; = i); clearly, no older samples X; with 
k <n — 2 affect the MA Xn. 
Alternatively, an order 1 autoregressive (AR) estimate could be used, i.e., for n > 1, 


Xn = aXn-1+(l—a)Xn 
=> Xy—-Xn-1 = (1-—a)(Xn—Xn-1); (1.20) 


where 0 < a < 1 is the "forgetting factor" and Xo = Xo. Note that all past values of X 
contribute to the current value of this autoregressive processes according to weights that 
exponentially diminish: 


Xn = a "Xo +(1—a)[a™ 1X, +a? X 4...+aXn-1 + Xn]. 


Also note that if 1 — a is a power of 2, then the autoregressive update (1.20) is simply 
implemented with two additive operations and one bit-shift (the latter to multiply by 1 — a). 
There is a simple trade-off in the choice of a. A small a implies that X„ is more responsive 
to the current samples X;,, but this can lead to undesirable oscillations in the AR process X. 
A large value of a means that the AR process will have diminished oscillations ("low-pass" 
filter) but will be less responsive to changes in the distribution of the samples X% [136, 164]. 


E EXAMPLE 1.9 


Consider a sequence of independent random variables X,,. Initially, the distribution 
is uniform on the interval [0,1] (i.e., EX = 0.5), but for n > 20 the distribution is 
uniform on the interval [3,4] (i.e., EX changes to 3.5). We see from Figure 1.5 that 
for forgetting factor œ = 0.2, a sample path of the first-order AR process X responds 
much more quickly to the change in mean (at n = 20) but is more oscillatory than 
the corresponding sample path of the AR process using forgetting factor a = 0.8. 
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Figure 1.5 Performance of first order AR estimators. 


1.13 MEASURES OF SEPARATION BETWEEN DISTRIBUTIONS 


Consider two cumulative distribution functions F; and Fz. Suppose we wish to measure 
the degree to which they are different. More formally, we wish to stipulate a kind of metric 
m( Fi, F2) € RT on the set of such distributions satisfying certain properties: 


e m(F\, F2) = 0 when F = Fo and 
èe m increases as the amount of "separation" between F; and F> increases. 


In networking, such metrics have recently been proposed (in, e.g., [67]) to detect anomalous 
activity. That is, suppose a CDF F; representing the nominal distribution of an attribute 
of a flow of packets (e.g., of their destination port number attributes [41]) is known. The 
CDF F; represents the corresponding distribution as measured online. When m/(F), F2) 
exceeds a certain threshold, the network operator (or some automatic intrusion detection 
system (IDS) ) may deem that the packet flow is exhibiting abnormal behavior and decide 
to take action (e.g., filter-out/firewall any identified offending packets). 

Some specific ways to measure the separation between two distributions include the 
Kolmogorov-Smirnov distance between two CDFs: 


mF, Fo) = max | (x) — F2(z)|. 


For a parameter a > 1, we can also define 


m(Fi, Fo) = ts |Fi(2) — Fa(x)|* a) 


ox 
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where we note that, as a — oo, this metric converges to that of Kolmogorov-Smirnov. 
Given two PMFs pı and p2 on the same state space we can define their chi-squared 
separation as 


(p(x) — a(x)? 
3 ple) 


A similar definition exists for PDFs, i.e., for continuous distributions. 
Given the means y and variances o of two distributions, the Fisher separation between 
them is 


|ui — bel. 

of +05 | 
alternatively, the standard deviation could be used instead of the variance in the denominator 
leading to a dimensionless Fisher separation metric. Note how the Fisher separation is 
increasing in the difference of the means but decreasing in either variance. There are other 
potentially significant features of a distribution besides the mean and variance, e.g., median 
and mode that can be used as a basis of a measure of separation ({106], p. 185). 

In [67], the entropy of a distribution was argued to be a significant feature for detection 

of distributed denial-of-service (DDoS) attacks. The entropy of a PMF p with strict range 
R is defined to be 


Y= p(x) log p(x). 
zER 


Thus one can consider the difference between the entropies of two distributions with the 
same range R, pı and p2, as a measure of their separation. Another useful measure of 
separation is the Kullback-Leibler distance [49]: 


eee. 
Em 


Again, similar definitions exist for PDFs. 


1.14 STATISTICAL CONFIDENCE 


Central limit theorems (CLTs) date back to Laplace and DeMoivre. They demonstrated that 
scaled sums of IID random variables of rather arbitrary distribution converge in distribution 
to a Gaussian. Convergence in distribution does not necessarily involve the existence of a 
limiting random variable and is a notion of convergence that is weaker than that of the weak 
LLN. A common use of the CLT is to evaluate the degree of "confidence" in the result of a 
group of experimental trials such as those obtained from a simulation study or a poll. 


1.14.1 A central limit theorem 


Suppose we have an IID sequence of random variables X1, X2, X3,.... Also, as in the case 
of the weak LLN, suppose that the common distribution has finite variance, i.e., 


o* = var(X) = E(X -EXP <œ. 
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where X ~ X; and ø > 0. Finally, suppose that the mean exists and is finite. i.e.. 
bh = EX <x. 
Define the cumulative sum 
Sn = Xi +X +e +X, 


forn > 1 and note again that ES,, = ny and var(S,,) = no?. Thus, for all n. 


Sn — nu O 
e( oyn ) m 


var PEN a 1 
ayn o 


and 


Theorem 1.14.1. Forall x € R, 
; Sn- nyu mw l 2y 
lim P { = <r) = ae /? dz. 
jmp (Pot sa) = f ge 


That is, 
Sn — ny 
oyn 


converges in distribution to a standard (mean 0 and variance |) Gaussian. A proof for this 
central limit theorem is given in Section 5.2 of [63]; see Problem 1.16 at the end of this 
chapter. 


1.14.2 Confidence intervals 


Suppose that n identically distributed samples X1, Xo. X3..... Xn have been generated 
by repeated trials of some experiment. Let sz anda > 0 be the mean and standard deviation, 
respectively, of Xg. Assuming a central limit theorem for the sequence {X} }. we have that 


vic, =u) (1.21) 





is approximately distributed as a standard Gaussian random variable, where the sample 
mean (or empirical mean) is 


1 n 
Xn meee 
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Figure 1.6 PDF of a standard Gaussian random variable exp(—z?/2)//2r. 


The mean su is taken to be an unknown quantity that is to be estimated from the samples 
Xx. Note that the sample mean is an unbiased estimate of j, i.e., 


EX, = p. 


This, in turn, allows us to assume that the quantity in (1.21) is Gaussian distributed for 
relatively small n and use this to compute "error bounds" on the law of large numbers. If 
Y is a standard Gaussian random variable, 


P(Y| <2) = (2) - &(-2) 
f 1 e7?"/2 d 
= — z 
—2V 20 
= 0.95; 


see Figure 1.6. Assuming that the standard deviation o is known, by the central limit 
approximation, for all sufficiently large n, 


Vn = ae 26 == 20 
0.95 = P [| —(Xn - <2)=P Xn - =, Xn + =} }. 1.22 
( Bak p)| < HE T $ Tr (1.22) 
So, with probability 0.95 (i.e., "19 times out of 20"), the true mean j resides in the interval 
= 20 = 20 
Xn — =, Xn + =]. 
| ral z 


Consequently, this interval is called the 95% confidence interval for p. 
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Typically, in practice, the standard deviation g is also not known and must also be 
estimated from the samples X;,. The sample variance is 


ani Ti me 
2 aN, (Xp, Sa 


: (: Sa) <a 


The implicit assumption is that a more general form of central limit theorem holds when 
the sample standard deviation, 


On = a2 > 0. 


is used instead of the true, but unknown, standard deviation ø above [194]. That is, the 
95% confidence interval for jz is taken to be 


[x IG os ots =| (1.23) 


Xn - =, Xn ay aa 
Va i 


It turns out that the sample variance defined above is a consistent but biased estimator 
for o°. In fact, 


= z 
Ec? = r (1.24) 


Thus, for a small number n of samples, it may prove more accurate to use 


n o2 
n-1” 





instead of g,, in the 95% confidence interval formulation above. That is, 


n -= 
2 
on 





n-1 


is both a consistent and unbiased estimator of o°. Note that this quantity and oe are both 
consistent estimators of o°, i.e., by the LLN, they both converge to a” as n — x. 

When the number of available samples n is small (less than 30), the quantity in (1.21) 
approximately follows a Student’s ¢ distribution [106]. Thus, instead of (1.22), the following 
is used to define the confidence interval for small sample sizes: 


p (cx. - 1) < (0.95) } = 0.95. (1.25) 


where the function ¢,, is defined by the Student’s ¢ distribution with n degrees of freedom 
(samples). 

Confidence intervals are discussed in [155, 194], Section 5.3 of [63], and Section 2.3 of 
[74]. 
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1.14.3 Recursive formulas and a stopping criterion 


In terms of simulation code, a simple recursive formula in the number of observed samples 
for the sample mean and sample standard deviation may be convenient [194]. For the 
sample mean, we clearly have 


= 1 
Xa = —((n-1)Xn-1 + Xn). 


Also, the sample variance satisfies 





= = 1 a= woes 
oz = Ct AS (-07n-1 + (Xn Ez Xn)?) . (1.26) 
n 
Let 
= Ae 
n = >TO 


be the unbiased estimate of the variance. From (1.26), one can derive the following identity: 


te = a ea + Xn ~ Kal? (1.27) 
Recall that a basic assumption in the computation of the confidence interval above was 
that the sample points X; are independent. In practice, a simulation may produce sequences 
of highly dependent X;. In this case, several independent batches of consecutive samples 
may be obtainable by simulation [106]. The confidence interval is taken using the sample 
mean and standard deviation of each batch; note that the "straight" sample mean X,, would 
equal that obtained from the batches. 
Finally, in order to arrive at a criterion to terminate a simulation, define the relative error 


‘, = var(X n) 1.28) 
"Xa 
on 
= ——. (1.29) 
Vn|X nl 


Here we have assumed u # 0 and o? < oo. A simulation may be terminated when the 
relative error reaches, say, 0.1, i.e., the variance of the sample standard deviation is 10% of 
the sample mean. Note that we can express the statement of the confidence interval (1.23) 
as 


P (+ [=ë L+ 2n]) = 0.95, (1.30) 
Xn 

where we could use Çn (0.95), defined using the Student’s ¢ distribution in (1.25), instead of 

"2" in (1.30) if the number of samples n is small. So, using the stopping criterion £n < 0.1, 

the claim is then made that the sample mean Xn is accurate to within 20% of the true mean 

with 95% probability. Also, for a relative error of 0.1, the number of required samples is, 


by (1.28), 
F 2 o 2 
n = 100 (Ze) = 100 (z) A (1.31) 
Xn H 
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In summary, a simulation produces IID samples X; from which a sequence of sample 
means X, and sample variances c2, are computed. The simulation may be terminated 
when the computed relative error Ên reaches a predefined threshold. The accuracy of the 
resulting estimate X „ of the quantity of interest jz can be interpreted in terms of a confidence 
interval (implicitly invoking a central limit theorem or Student’s t distribution). 


E EXAMPLE 1.10 


A great deal of current research activity in networking involves comparisons of the 
performance of competing mechanisms (devices, algorithms, protocols, etc.) by 
simulation or through prototypical deployment on the Internet. Suppose that n trials 
are conducted for each of two mechanisms in order to compare their performance, 
leading to a dataset 


{Dk i} 


for mechanism k € {1.2}. Also suppose that, for all ¿, the ith trial (yielding data 
Dı; and D3) was conducted under arguably equal environmental conditions for 
both mechanisms. Let 


Xi = Dia- Dai 


be the difference in the performance of the two mechanisms for the common environ- 
mental conditions of trial 7. That is, for each trial, an "apple-to-apple" comparison is 
made using coupled or paired observations. 

To assess whether EX > 0 (respectively, EX = 0), we simply compute the 
confidence interval according to (1.22) or (1.25) and determine whether the origin is 
to the left of it (respectively. contained by it). 

Given uncoupled observations that are different in number, a procedure for deciding 
whether EX = 0 (i.e., a "t test") is summarized in Section 13.4.2 of [106]. 


1.15 DECIDING BETWEEN TWO ALTERNATIVE CLAIMS 


Suppose that a series of measurements X; are drawn from the Internet. A problem is that the 
Internet may be in different "states" of operation leading to samples X; that are independent 
but not identically distributed. As a great simplification, suppose that the network can be 
in only one of two states indexed by j € {1,2}. Also suppose that a batch of n samples 
Xi, 1 < į <n, is taken while the network is in a single state but that state is not known. 
Finally, suppose it is known that the probability that the network is in state 1 is pı and that 


ft; = E(X; | network state j) 
for known values uj, j E {1,2}. Without loss of generality, take zı < p2. 


We wish to infer from the sample data X; the state of the network. More specifically, 
we wish to minimize the probability of error Pe in our decision. To do this, note that by the 
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central limit theorem, given that the network is in state j, the sample mean 


= 1 


is approximately normally distributed with mean jz; and variance o? n: An unbiased, con- 
sistent estimate of this variance is 


ana 1 ik a, 
ao, = So (Xi — Xn)’. 
=1 


n—-1¢ 
a 





To determine the probability of decision error, we will condition on the state of the 
network: 
Pe = P(error | network state k) pp. (1.32) 


2 
=1 


k 


where pọ = 1 — pı. To this point, we have not described the method by which a decision 
on the network state is made. Consider a decision based on the comparison of the sample 
mean X,, with a threshold 6, where 


fa SO < po, 


so that the network is deemed to be in state 2 (having the higher mean) if X, > 8; 
otherwise, the network is deemed to be in state 1. So, a more concrete expression for the 
error probability (1.32) ensues because we can approximate 


P(error | network in state 2) = P(X, < 8 | network in state 2) 


-0 
x ð H2 , 
a3 








‘n 


where Ẹ is 1 minus the CDF of a standard Gaussian distribution. Using a similar argument 
when conditioning on the network being in state 1, we arrive at the following expression: 








8 — —0 
E |p. +0 | Z 
V Cis GS 


Thus, to determine the optimal value of 8, we need to minimize P, over 6. This approach can 
be easily generalized to make decisions among more than two alternative (mutually exclu- 
sive) hypotheses. In [113], Wald’s classical framework to decide between two alternative 
claims based on sequential independent observations is used on failed scan (connection 
set-up attempt) data to detect the presence of Internet worms. 


P =ò p2- (1.33) 


Problems 


1.1 For any two events A and B show that 


P(A) < P(ANB)+P(B). 
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1.2 Prove the law of total probability (1.1). 
1.3 Consider two independent random variables X; and X2. 
(a) Show that they are uncorrelated, i.e., cov( X1, X2) = 0. 
(b) Show that var( X; + X2) = var(X,) + var( X2). 
1.4 Suppose that X is a continuous random variable uniformly distributed on the interval 


{-1,1] and that Y = X?. Show that X and Y are uncorrelated but that they are (clearly) 
dependent. 


1.5 Two random variables Xı and X% are said to be jointly Gaussian distributed if their 
joint PDF is 





1 
fx, X2(£1: £2) On det(C) exp( (x pb) Ce = u)) f (1.34) 
where the (symmetric) covariance matrix is 
= var( X1) cov( X1, X2) _ | EX 
cov( X1. X2) var( X2) : “| EX 


and det(C) is the determinant of C. Show that if two jointly Gaussian random variables 
are uncorrelated, then they are also independent.’ 


1.6 For jointly Gaussian X; and X2 show: 


(a) For scalars a, and a2, a} Xı + a9X9 (a "linear combination" of Gaussian random 
variables) is Gaussian distributed. 


(b) E(X|Y’) is Gaussian distributed. 
1.7 Show that a Bernoulli random variable X with Rx = {0,1} can always be repre- 


sented as an indicator function 14 for some event A (express A in terms of X). Prove 
Jensen’s inequality (1.13) for a Bernoulli random variable. 


1.8 Prove Jensen’s inequality for any discretely distributed random variable. 


1.9 Find the MGF of a geometrically distributed random variable and verify the expression 
given for the MGF corresponding to a Poisson distribution. 


1.10 Compute the variance of the distributions described in Sections 1.4 and 1.5. 


1.11 IfP(X > 0) =1(e., X is nonnegative almost surely), show 


ex= P(X > x) dex. 
o 


1.12 Suppose X is Cauchy distributed with PDF 


1 


T= ra) 


The joint PDF of n > 2 jointly Gaussian random variables has the same form as (1.34) except that the 27 term 
is replaced by the more general term (27)®/2, 
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forz ER. 
(a) Verify that E(X1{X > 0}) = œ and that E(-X1{X < 0}) = æ 
(b) Conclude that EX does not exist. Note that the question of existence is different from 


the question of finiteness. 


1.13 The linear least square error (LLSE) estimator of a random variable X given a 
random variable Y is a linear function h(Y) = aY + b which minimizes the MSE 


EX — A(Y))?}; 


i.e., for the LLSE estimator, the constants a and b are chosen so as to minimize the MSE. 
Show that the LLSE estimator is the conditional expectation E(X|Y) when the random 
variables X and Y are jointly Gaussian with var(Y ) > 0 (so, in this case, the best estimator 
is a linear one). 


1.14 For the gamma distribution, verify (1.7) and (1.8) and show that 
rT, = (r-1)! 
when r € ZT. 


1.15 Use MGFs to prove that the binomial distribution with parameters n and q converges 
to the Poisson distribution with parameter À as q — 0 and n — œ in such a way that 
nq — à. This is Poisson’s theorem (sometimes called the law of small numbers). Hint: 
(+r) seasz| 0. 


1.16 Consider a random variable Y, that is Poisson distributed with parameter A, i.e., 
EY) = à. 


(a) Show that var(Y)) = À too. 
(b) Using MGFs, show that, as \ — oo, the distribution of 


Yy -à 
Vd 


converges to a standard (mean 0 and variance 1) Gaussian distribution. 





This is a kind of CLT for Poisson processes; see Chapter 2. The CLT has been generalized to 
many other contexts, including functional CLTs on the convergence of stochastic processes 
to Brownian motion, see [21]. 


1.17 Prove (1.24). 
1.18 Prove (1.26) and (1.27). 


1.19 If U is a continuous random variable that is uniformly distributed over (0, 1], show 
that F~'(U) has CDF F, where 


F(u) = inf{x| F(z) =u}. 


1.20 Suppose a group of (distinct) persons each has an independent birthday among the 
365 possible ones. Determine the minimum number n of such persons required so that the 
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probability that at least two share a birthday is greater than or equal to 0.5. 

Hints: Compute the probability of the complementary event and the answer is between 20 
and 25. 

The surprisingly small result is known as the birthday paradox. 


1.21 Suppose a hacker wishes to have a domain name server (DNS) associate the domain 
name www.kesidis.com with one of his own 32-bit Internet Protocol (IP) addresses so that 
he can intercept some of the critically important correspondences that are directed to this 
site. The hacker simultaneously transmits q identical queries for www.kesidis.com to the 
targeted DNS. Further suppose that the targeted DNS naively forwards each query to an 
authoritative DNS using IID transaction identifiers (used to authenticate the authoritative 
DNS’s response) which are 16 bits long and not known to the hacker. Shortly thereafter, 
and before the authoritative DNS can reply, the hacker also transmits s responses to the 
targeted DNS spoofing those of the authoritative DNS, where each such response associates 
www.kesidis.com with the hacker’s chosen IP address and contains a guess at one of the 
transaction identifiers generated by the targeted DNS. Assuming 


s = qQ] En, 


find the value of n so that the probability that a forwarded query and spoofed response have 
the same transaction identifier is 0.5, i.e., the probability that the hacker guesses correctly 
and thereby poisons the targeted DNS’s cache. 


1.22 In an idealized network using ALOHA medium access control, each of n nodes 
attempts to transmit a packet in a time slot with probability p, after the initial packet 
transmission failed due to interference from another host. Such retransmission decisions 
are independent. Suppose all hosts are synchronized to common time slot boundaries and 
that they are always in "retransmission" mode (with a packet to transmit). 


(a) Find the probability that a packet is successfully transmitted in a given time slot by 
a given node. 


(b) Find the probability that a packet is successfully transmitted in a given time slot by 
any node. 


(c) Find the expected number of successfully transmitted packets per time slot by the 
group of nodes, i.e., the network’s throughput. 


(d) Show that the throughput is maximized by the choice of p = 1/n and that, as n — œœ, 
the throughput converges to 1/e ~ 0.37 packets per time slot. 


(d) Show that the maximum throughput of unslotted ALOHA is 1/(2e). 


1.23 Show that the minimizing value of the decision threshold 8 in (1.33) is (441 + 2) /2 
when the sample variances are equal, i.e., of , = 03. 


1.24 Consider a link carrying packets to a Web server. Suppose that, under normal 
operating conditions, the link will subject the Web server to a data rate of 4 Mbps. However, 
when the Web server is under a DDoS attack, the link will carry an average of 6 Mbps to the 
server. AnIDS samples the link’s data rate and determines whether the server is under attack. 
Assume known standard deviations in the data rate of 1 Mbps under normal conditions and 
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of 1.5 Mbps under attack conditions. Finally, assume attack conditions exist 25% of the 
time. Find the value of the optimal decision threshold @ (compared against the sample 
mean) that minimizes the probability of decision error. 


1.25 In the previous problem, instead of minimizing the probability of decision error, 
suppose the probability that it was decided that the network is under attack when, in fact, it 
was not can be no more than 0.10, i.e., 


P,,(@) = P(decision error | no attack) < 0.10. 


Again, this decision is based on the sample mean. Note that this event is called a false 
positive or type lI error. Such false positives can be caused by legitimate "flash crowds" of 
temporary but excessive demand. Subject to this bound we wish to minimize the probability 
of missed detection ("type I" error) of an actual attack: 


Pa(0) = P(decision error | attack). 
That is, find 


ar min Pa(0). 

8 | Pa (0) <0.10 

Tbis is called a Neyman-Pearson test [174] (of the sample mean with the computed value 
of 8). 


1.26 Consider a simple binary symmetric channel model wherein a single bit X € {0,1} 
is transmitted and a single bit Y is received such that the transmission error probabilities 
are equal, i.e., P(Y = 1|X = 0) = P(Y = O|X = 1) = ce. Fori € {0,1}, find 
P(X = i|Y = i) (i.e., the probability that bit i was transmitted given bit 7 was received) in 
terms of ce and so = P(X = 0). 


1.27 Suppose that the reported outcome of a poll regarding a presidential race is, "Candi- 
date X will collect 48% of the vote with an error of +2% 19 times out of 20." Explain this 
outcome in the terms previously used to describe statistical confidence based on the CLT. 
Also, what are the basic underlying assumptions about which specific individuals, among 
the entire voting population, were polled for this statement to hold? 


1.28 Inthecontext of Example 1.10, suppose that we have compiled only the separate em- 
pirical distributions pı and p2, respectively, based on the data sets {D, ;}"_, and {Do;}7_), 
but not the empirical distribution of X = Dı — D2 on a trial-by-trial basis. Given pı and 
p2: 


(a) Can we obtain the empirical mean X,, of X? 


(b) Can we obtain the empirical variance of X ? 


CHAPTER 2 
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A stochastic (or random) process is a collection of random variables indexed by a parameter 
in a domain. In this book, this parameter will be interpreted as time (stochastic processes 
describing images can be indexed by more than one spatial parameter and those describing 
video will have joint spatial and temporal indexes). If the time parameter takes values only 
in Z* (or any other countable subset of R), the stochastic process is said to be discrete time; 
i.e., 


{X(t)|te Zt} 


is a discrete time stochastic process when X (t) is a random variable for all t € Z+. If the 
time parameter t takes values over R or R* (or any real interval), the stochastic process is 
said to be continuous time. 

The dependence on the sample w € Q can be explicitly indicated by writing X.,(t). For 
a given sample w, the random object mapping t > X.,,(t), for all t € RĦ say, is called a 
sample path of the stochastic process X. 

The state space of a stochastic process is simply the union of the strict ranges of the 
random variables {X (t) | t € Z+}. In this book, we will restrict our attention to stochastic 
processes with countable state spaces. In this chapter and for the majority of the examples 
that follow, we will take the state space to be Z, ZY, or a finite subset {0, 1, 2, ..., K}. Of 
course, this means that the random variables X (+) are all discretely distributed. 

Most of the queuing systems studied in this book are substantially simpler to analyze 
when they are considered operating in continuous time. So, we focus our attention on 
continuous-time Markov chains in this chapter. Discrete-time Markov chains are briefly 
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covered at the end of the chapter. Refer to [20, 118, 119, 161, 193] for further details 
concerning Markov processes. 


2.1 MEMORYLESS PROPERTY OF THE EXPONENTIAL DISTRIBUTION 


In this section, the salient properties of exponentially distributed random variables are 
discussed. These properties are fundamental to the subsequent construction of Markov 
chains. 


Theorem 2.1.1. Jf X is exponentially distributed, 


P(X >z+y| X >y)=P(X > 2). 


This is the memoryless property and its simple proof is left as an exercise. For example, 
if X represents the duration of the lifetime of a light bulb, the memoryless property implies 
that, given that X > y, the probability that the residual lifetime (X — y) is greater than z is 
equal to the probability that the unconditioned lifetime is greater than x. So, in this sense, 
given X > y, the lifetime has “forgotten” that X > y. Only exponentially distributed 
random variables have this property among all continuously distributed random variables 
(see Problem 2.3 at the end of the chapter) and only geometrically distributed random 
variables have this property among all discretely distributed random variables. 

Now suppose that X, and X2 are independent and exponentially distributed random 
variables with parameters A; and Ag, respectively. The following two properties will be 
fundamental to our discussion of Markov processes in this chapter. 


Theorem 2.1.2. min{ X1, X2} is exponentially distributed with parameter A, + Av. 


Proof: Define Z = min{X;, X2} and let Fz, F), and F> be the CDF of Z, X1, and X3, 
respectively. Clearly, Fz (z) = 0 for z < 0. For z > 0, 
1—Fz(z) = P(min{X,.X2} > z) 
= P(X, > z,X2 > 2) 
= P(X, > 2z)P(X2 >z) by independence 
= exp(—(A1 + A2)z) 


as desired. o 


Theorem 2.1.3. 


Ay 
Ay t Ao 





P(min{ X1, X2} = Xı) = 
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Proof: 
P(min{ X1, X2} = X)) = P(X, < Xə) 
o £2 
= I / Ae"**1 dz, Age 77"? dao 
=œ J -oœ 
= 1. Age Oi tA2)#2 dro 
=00 
cen Ai 
~ Mty 
as desired. o 


Two independent geometrically distributed random variables also have these properties. 


2.2 FINITE-DIMENSIONAL DISTRIBUTIONS AND STATIONARITY 


Consider a stochastic process 
{X(t) |t€R*} 


with state space Z*. Let Pi „ta... tn be the joint PMF of X (t1). X (t2), ..., X (tn) for some 
finite n and different ty € Rt forall k € {1,2,...,n},ie., 


Pty tosectm(€1502,-,2n) = P(X (t1) = z1, X (t2) = vo,..., X (tn) = Ln). (2.1) 


This is called an n-dimensional distribution of X. The family of all such joint PMFs is 
called the set of finite-dimensional distributions (FDDs) of X. Note that the FDDs are 
consistent in that one can marginalize (reduce the dimension) and obtain another, e.g., 


Pty tata (Z1: Z2,24) = 5 Pty tats talis 02, 03,04). (2.2) 
£3EZ+ 


Beginning with a family of consistent FDDs, Kolmogorov’s extension (or "consistency”) 
theorem [104] is a general result demonstrating the existence of a stochastic process t — 
X,,(t), w € Q, that possesses them. In the following sections, we will begin with a sample 
path definition of a Markov process and, from that, show how to derive its FDDs. 

A stochastic process X is said to be stationary if all of its FDDs are time-shift invariant, 
i.e., if 


Ptystangtr = Ptitrtetr....tn tT (2.3) 


for all integers n > 1, all ty € R*, and all r € R such that tg + 7 € Rt forall k. 
A weaker definition for stationarity is wide-sense stationarity (WSS), which only re- 
quires: 


(i) EX(t) is a constant function of t and 
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(ii) the autocorrelation E[X (t)X(s)| depends on t and s only through their difference 
t—s,1e., 


ELX(t)X(s)} = E[X(t+7)X(s+7)] = R(t-s) 
for all 7, where R is the autocorrelation function of X. 


The WSS processes are important in communication theory wherein signals are corrupted by 
interference and noise (in particular, additive broadband white noise having autocorrelation 
function R = 4, the Dirac delta function). Such signals can be filtered by linear time- 
invariant systems and analyzed in the frequency domain. The Fourier transform of a WSS 
signal’s autocorrelation is its power spectral density [164]. Queues are, however, highly 
nonlinear systems so we will not consider further the class of WSS signals in this book, but 
see [52] for an interesting application of linear systems theory to queues. 


2.3 THE POISSON (COUNTING) PROCESS ON Rt 


A counting process X on R? is characterized by the following properties: 


(a) X has state space Z*, 
(b) X has nondecreasing (in time) sample paths that are continuous from the right, i.e., 


pore) = X(s), and 


(c) X(t) < X(t—) + 1 so that X does make a single transition of size 2 or more, where 
t— is a time immediately prior to t, i.e., 


X(t-) = lim X (s). 


For example, consider a post office where the ith customer arrives at time T; € Rt. 
We take the origin of time to be zero and, clearly, T; < Ti+ı for all i. The total number 
of customers that arrived over the interval of time [0, t] is defined to be X(t). Note that 
X(T;) = i, X(t) < cif t < Tj, and X(t) — X(s) is the number of customers that have 
arrived over the interval (s, t]. In other words, 


x 
X(t) = $0 HT; < t} = max{i| T; < t} 
i=1 
Of course, X is an example of a continuous-time counting process that is continuous from 
the right. For a sample path of X see Figure 2.1. 
Consider the sequence of interarrival times S; = T; — T;—1ı for i € {1, 2,3, ...}, where 


To = 0. 


A Poisson process is a continuous-time counting process whose interarrival times {5;}32, 
are mutually IID exponential random variables. Let the parameter of the exponential dis- 
tribution of the S;’s be A, i.e., ES; = AT! for all i. Since 
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Figure 2.1 A sample path of the Poisson process. 


Tn is Erlang (gamma) distributed with parameters À and n; see Section 1.9.1. 

A natural question to ask about a stochastic process X is: What is the distribution of 
X(t) for some specified time t? In other words, what is the (one-dimensional) marginal 
distribution of the process? We will show that for the Poisson process, X(t) is Poisson 
distributed with mean At. For this reason, À is sometimes called the intensity (or "mean 
rate" or just "rate") of the Poisson process X. 


Theorem 2.3.1. X(t) is Poisson distributed with parameter Xt. 


Proof: First note that, for t > 0, 


Now, for an integer i > 0 anda real t > 0, 
OO \it] pig—Az 
P(X() <i) = Pan = | AT 


f i! 


where we have used the gamma PDF; see Equation (1.18). By integrating by parts, we get 


; Nz ee Oo yizi-1g—Az 
ee = ene H c= 
(At)ie>* œ \i,i-1g-Az 
= FS yp ——,, d 
ee uen 
After successively integrating by parts in this manner, we get 

At) eT At)le-àt co 

P(X(t) <i) = CO eo BOE | eee 
i t 


—At 


t: 
5 (Ate 
= 
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Now note that {X (t) = i} and {X (t) < i — 1} are disjoint events and 





{X(t) =i} U{X() si-1} = {XH <4}. 
Thus, 
P(X(t) =i) = P(X(t) <t)-P(X(t) <i-1) 
tie SS (ate 
~ i] | 
jo T jo f 
7 (At)? —At 
~ i! 
which is the Poisson PMF as desired. o 
Now consider k disjoint intervals of real time (1, t1], ($2, ta], --. (Sk, tk] with 
Sı < ti < S2 <t < ++: < Sk < tk. (2.5) 
Here, X is a Poisson process if and only if, for all k, all s,,t1, s2,t2, ... ,Sk:-tk € Rt 


satisfying (2.5), and all n1, n2, ...np € Zt, 
oe 


- (pace Alti = SP MG), (2.6) 


=1 


Note that X (t;) — X (si) (which is called an increment of X ) is the number of transitions of 
the Poisson process in the interval of time (s;, t;]. Thus, (2.6) states that the Poisson process 
has independent (nonoverlapping) increments. Also, the increment over a time interval of 
length 7 is Poisson distributed with parameter A7 (a slight generalization of Theorem 2.3.1). 
The Poisson process is the only counting process that 


e has Poisson distributed independent increments, or 
e has IID exponentially distributed interarrival times, or 
@ possesses the conditional uniformity property [55, 183]; see Problem 2.14. 


That is, all of these properties are equivalent. The memoryless property of the exponential 
distribution is principally responsible for the independent increments property of a Poisson 
process, as will be argued in a more general Markovian context in the next section. 

We will now drive the k-dimensional marginal distribution of a Poisson process by using 
the independent increments property. So, consider times 0 < tı < t2 < +- < ty and 


P(X (t1) = mi, X (t2) = M2, a X (tk) = Mk), (2.7) 


where m1, M2, ..., Mp € Zand m; < mi+ı forall į (otherwise the probability above would 
be zero). Defining Am; = m; — m;_1 and AX; = X (ti) — X(t;_1), the quantity in (2.7) 
equals 
P(X (t1) = mi, AX? = Amo. te., AXk = Amg) 
= P(AX2 = Amo, teg AX; = Amr | X (t1) a mı) P(X (ty) = mı) 
P(AX2 = Amo, aid AX; od Amg) P(X (t1) = mı), 
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where the last equality is by the independent increments property. By repeating this argu- 
ment, we get that the quantity in (2.7) equals 


k 
P(X (ti) = mi) [[ P(X (ts) — X (6-1) = mi — mi-1) 


1=2 


m k Mi -Mi— 
(At, )™ enh Il ae ti-1)) T l eTAlti—tia) 
Mi — Mi—1}: 





mı! rae i 


which is in agreement with (2.6). 


2.4 CONTINUOUS-TIME, TIME-HOMOGENEOUS MARKOV PROCESSES 
WITH COUNTABLE STATE SPACE 


We will now define a kind of stochastic process called a Markov process of which the Poisson 
process is an example; more specifically, a Poisson process is a variety of (transient) pure 
birth Markov process. A Markov process over a countable state space © is sometimes called 
a Markov chain. Without loss of generality, we will subsequently take © = Z+. A Markov 
chain is a kind of random walk on ©. It visits a state, stays there for an exponentially 
distributed amount of time, then makes a transition at random to another state, stays at this 
new state for an exponentially distributed amount of time, then makes a transition at random 
to another state, etc. All of these visit times and transitions are independent in a way that 
will be more precisely explained in the following. 


2.4.1 The Markov property 


Definition 2.4.1. If, for all integers k > 1, all subsets A, B, B,,..., Bk C È, and all times 
t, 8,81, .., Sk E RT such that t > s > s1 > > Sx, 


P(X(t) € A| X(s) € B, X(s1) € Bi, «+: , X (sk) € By) 
= P(X(t) € A| X(s) € B), (2.8) 


then the stochastic process X is said to possess the Markov property. 


If we identify X (t) as a future value of the process, X (s) as the present value, and past 
values as X (s1), ..., X (sx), then the Markov property asserts that the future and the past 
are conditionally independent given the present. In other words, given the present state 
X(s) of a Markov process, one does not require knowledge of the past to determine its 
future evolution. Any stochastic process (on any state space with any time domain) that 
has the Markov property is called a Markov process. As such, the Markov property is a 
"stochastic extension” of notions of state associated with finite-state machines and linear 
time-invariant systems. 
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The property (2.8) is an immediate consequence of a slightly stronger and more succinctly 
stated Markov property: for all times s < ¢ and any (measurable) function f, 


E(f(X:)|X,,0<r<s) = E(f(Xz)| Xs). (2.9) 


2.4.2 Sample path construction of a time-homogeneous, continuous-time 
Markov chain 


In this section, how a Markov chain X makes transitions from state to state will be described. 
For a time-homogeneous Markov chain, consider each state n € Zt and let 


1 


—n.n 





> 0 


be the mean visiting time of the Markov process, i.€., qn,n < 0. That is, a Markov chain is 
said to enter state n at time T and subsequently visit state n for S seconds if X (T—) £ n, 
X(t) =nforallT <t<S+T,and X(S+T) #n. Also, define the assumed finite set 
of states 


Ta GE ny 


to which a transition is possible directly from n. For all m € Tn, define gqn,m > 0 such that 
the probability of a transition from n to m is 


n,m 


an,n 


> 0. 


Thus, we clearly need to require that 


Y -B® = 1, 


mETn Ann 


ie, forall n € Z*, 


Ses St (2.10) 


mEZt 


where qn,m = 0 for all m ¢ T U {n}. 
Now let T; be the time of the ith state transition with To = 0, i.e., the process X is 
constant on intervals |[T;—1. T;) and 


X(T) = X(Ti-) # E 


for all i € Z*. Let the column vector (0) represent the distribution of X (0) on Z*, so 
that entry in the nth row is 


Tn(0) = P(X(0) =n), 


i.e., m(0) is the initial distribution of the stochastic process X. 
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An alternative description of how a Markov chain X with time domain R” makes 
transitions from state to state will now be given. Suppose that X(T;) = n € ZT. To 
the states m € Tn, associate an exponentially distributed random variable $;(n.2m) with 
parameter g;.m > O (recall this means ES;(n.m) = 1/qn.m). Given X(T;) = n, the 
smallest of the random variables 


{Si(n,m) |m € Ta} 


determines X (T;+1ı) and the intertransition time T;+ı — T;. That is, X(T;-1) = j if and 
only if 


Tai- T; = Si(n.j) = min S;(n.m). 


mETn 


The entire collection of exponential random variables 
{Si(n.m) lice Z* ne Z* me Ta} 


are assumed mutually independent. Therefore, by Theorem 2.1.2 and (2.10), the intertran- 
sition time T;., — T; is exponentially distributed with parameter 





~inn = 5 anm: (2.11) 
meT, 
so that 
` 1 
E(Ti41-Ti) = > 0 
—Gnn 


in particular. Also, the state transition probabilities 
P(X (Tigi) =j9 | X(T;) =n) = P(Si(n,j) = min Si(m.m)) 


= . (2.12) 
n,n 


where the last equality is by Theorem 2.1.3 and definition (2.11). Note again that if a 
transition from state n to state j is impossible (has probability zero), gn j = 0. 
In this book, we will always assume that 


—Gnn < æ forall states n, 


i.e., the Markov chain is conservative. Also, we have assumed that the Markov chain is 
temporally (time) homogeneous, i.e., for all times s, t > 0 and all states n, m: 


P(X(s+t)=n|X(s)=m)=P(X(t)=n|X(0) =m). (2.13) 
In summary, assuming the initial distribution 7(0) and the parameters 
{anm | nm E Zy 


are known, we have described how to construct a sample path of the Markov chain X from 
a collection of independent random variables 


{Sin m) | iE Zt, ne X= Zt, me Th}, 
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Figure 2.2 A sample path of a Markov chain. 


where S; (n,m) is an exponentially distributed with parameter qn,m. When a Markov chain 
visits state n, it stays an exponentially distributed amount of time with mean —1/qn,n and 
then makes a transition to another state m € 7„ with probability —qnm/@nn- 

To prove that the processes thus constructed are Markovian (i.e., satisfy the Markov 
properties (2.8) and (2.9) ), letn = X(s) and let i be the number of transitions of X prior to 
the present time s. Clearly, the random variables 2, n, and T; (the last transition time prior 
to s) can be discerned from {X,, 0 < r < s} and can therefore be considered "given" as 
well. The memoryless property of the random variable T; 1 — T;, distributed exponentially 
with parameter —gp,n, implies that 


Pui -8 > z | Tini — T; > s — Ti) 
= P(T =T; > 2+ (s -T;)| Tim -Ti > s — Ti) 
= P(Ti+1 - T; > x) 
= exp(GnnZ) 


for all x > 0; see Figure 2.2. Note that exp(¢n,nz) depends on {X,, 0 < r < s} only 
through n = X(s). So, T;41 — s is exponentially distributed with parameter —qn,n and 
conditionally independent of s — T; given {X,, 0 < r < s}. Furthermore, {X,, 0<r< 
T;} is similarly conditionally independent of {X,, r > Ti+ı} given X(s) = n (by the 
assumed mutual independence of the {S;(m,m)} random variables ). 

Since the exponential distribution is the only continuous one that possesses the memo- 
ryless property,! one can show the converse statement, i.e., that the previous constructions 
are implied by (2.8) and (2.9). 


l! Again, the proof of this statement is left as an exercise; see Problem 2.3 at the end of the chapter. 
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@ EXAMPLE 2.1 


Clearly, a Poisson process with intensity À is an example of a Markov chain with 
transition rates 


à ifm=n+l1, 
n.n = —X if M = nr, 
0 else. 


2.4.3 The transition rate matrix and transition rate diagram 


The matrix Q having qnm as its entry in the nth row and mth column is called the transition 
rate matrix (or just "rate matrix") of the Markov chain X. Note that, by (2.10), the sum of 
the entries in any row of the matrix Q equals zero. The nth row of Q corresponds to state 
n from which transitions occur, and the mth column of Q corresponds to states m to which 
transitions occur. For n Æ m, the parameter qn.m is called a transition rate because, for 
any i € Z*, ES;(n, m) = 1/qn,m. Thus, we expect that, if qnm > qn j, then transitions 
from state n to 7m will tend to be made more frequently (at a higher rate) by the Markov 
chain than transitions from state n to j. 


@ EXAMPLE 2.2 


For example, the transition matrix of a Poisson process with intensity À > 0 is 


=e à 000 
Gd Be 00 
Q=| 0 0 -r’ 2X 0 


E EXAMPLE 2.3 


As another example, suppose the strict state space of X is {0, 1, 2} and the rate matrix 
is 


-5 2 3 
Q= 0 -4 4 (2.14) 
1 0 -i 


Note how we use just a 3 x 3 rate matrix Q since the strict state space is just a 3-tuple 
{0, 1,2} rather than all of the nonnegative integers, Z*. Similarly, the stationary 
distribution m would be a three-dimensional vector. Also note that, in this example, 
a direct transition from state 2 to state 1 is impossible (as is a direct transition from 
state 1 to state 0). Also, each visit to state 0 lasts an exponentially distributed amount 
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À À À À 


Figure 2.3 The transition rate diagram of the Poisson process. 


qo2 = 3 





G20 = 1 


Figure 2.4 A transition rate diagram on state space {0, 1, 2}. 


of time with parameter 5 (i.e., with mean 0.2); a transition to state 1 then occurs with 
probability 2 or a transition to state 2 occurs with probability 2. 


We can also represent the transition rates graphically by what is called a transition rate 
diagram. The states of the Markov chain are circled and arrows are used to indicate the 
possible transitions between states. The transition rate itself labels the corresponding arrow 
(transition). The transition rate diagrams of the previous two examples are given in Figures 
2.3 and 2.4, respectively. When an arrow is drawn and labeled with a transition rate q, the 
transition rate is assumed to be strictly positive. 
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2.4.4 The Kolmogorov equations 


Consider again the Markov chain X with rate matrix Q. For r € R* and n,m € Zt, 
define 


Pam(t) = P(X(s+7)=m|X(s) =n). 


Note again that we are assuming that the chain is temporally homogeneous so that the 
right-hand side of the above equation does not depend on time s. The matrix P(T) whose 
entry in the nth row and mth column is Pn,m (7) is called the transition probability matrix. 
Finally, for each time s € R+, define the sequence 


Tals) = P(X(s)=n) forn € Zt. 


So, the column vector 2(s), whose ith entry is 7;(s), is the marginal distribution of X at 
time s, i.e., the distribution (PMF) of X (s). 
Conditioning on X (s) and using the law of total probability (1.1), 


P(X(s+7) =m) sS PE (s+r)=m]|X(s)=n)P(X(s)=n) 
n=0 


for all m € Zt, i.e., 
Tm($ +7) = Ern m(T)Tn(s) for all m € Zt. 


We can write these equations compactly in matrix form: 
mi(st+tr) = 2«'(s)P(r), (2.15) 


where mT (s) is the transpose of the column vector z(s), i.e., zT (s) is a row vector. 
Moreover, any FDD of the Markov chain can be computed from the transition probability 
functions and the initial distribution. For example, for times t > s > r > 0, 
P(X(t) =n, X(s) =m, X(r) =k) 
= P(X(t)=n|X(s) =m, X(r) = k)P(X(s) =m, X(r) = k) 
= P(X(t)=n|X(s) =m)P(X(s) =m|X(r) =k)P(X(r) =k) 
= Dmn(t — 8)Pkm(s— r) XC P(X(r) = k | X(0) = i)P(X (0) = i) 


= Pm nlt- 5)Pkmls— r) » P(X(r) = k | X(0) = i)P(X(0) = i) 





= Pm,n t— S)Pk,m(S ji r) XO pi.e(r) 7: (0) (2.16) 


where the second equality is the Markov property. In the second-to-last expression, we 
clearly see the transition from some initial state to k at time r, then to state m at time s 
(s — r seconds later), and finally to to state n at time t (t — s seconds later). 

Our objective in the remainder of this section is to compute the transition probability 
matrix P(t) in terms of the known transition rate matrix Q. First note that 
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(a transition in an interval of time of length zero occurs with probability zero), which can 
be written in matrix form as 


P(0) = I, (2.17) 


where I is the (multiplicative) identity matrix, i.e., the square matrix with 1’s in every 
diagonal entry and 0’s in every off-diagonal entry. Now consider the quantity 


Pnm(é) = P(X(st+e)=m|X(s) =n) 


for n Æ m, a small amount of time 0 < £ < 1, and an arbitrarily chosen time s € R*. Let 
Vn be the residual holding time in state n after time s, i.e., X (t) = nforallt € [s,s+V,,) and 
X(s+V,) Æ n. The total holding time in state n is exponentially distributed with parameter 
—qn,n and therefore, by the memoryless property, Vn is also exponentially distributed with 
parameter —gn,,. Thus, 


Pnml(é) = P(V, Se) x == + of). (2.18) 


We will now explain the terms of the right-hand side. The first term represents the probability 
that the Markov chain X makes only a single transition (from n to m) in interval of time 
(s,s +€]. Recall that the probability that X makes a transition to state m from state n is 
—Gn,m/Gnn- The symbol o(£) ("little oh of £") represents a function satisfying 


ti =; 
e a0 E€ 


In (2.18), the term o(¢) represents the probability that the Markov chain has two or more 
transitions in the interval of time (s, s + £]; see Problem 2.5 at the end of this chapter. 
Now note that 


PV <e) = 1- eXp(EGn,n) 
= -€4n.n + 0(€), 
so that, after substituting into (2.18), we get 
Pnym(€) = Gn,mé + O(€), (2.19) 


which implies that 


n,m = n,m(O 
Pnim{E) = Pnm(0) _ nee. 
€ E 


where we recall that Pn (0) = 0 forall n Æ m. Letting € — 0, we get 
Pn,m(0) = drm; (2.20) 
where the left-hand side is the time derivative of Dy m at time 0. Finally, since 
Pn n(€) = l- 5 Pn,mlE), 
mEZt, mžn 


we get, after differentiating with respect to time, 


PopQ) = => dnm = qnn < 0, (2.21) 


meén 
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where we have used the definition of qnn. In matrix form, (2.20) and (2.21) are 
P0) = Q. (2.22) 


In fact, this statement can be generalized to obtain the Kolmogorov backward equations: 


Theorem 2.4.1. For all s > 0, 
P(s) = P(s)Q. (2.23) 


Proof: By (2.22) and (2.17), 
P(0) = IQ = P(0)Q. 
which gives the desired result for s = 0. For s > 0, take a real € such that 


0<¢< min{s, 1}. 


So, 
Pnim (8) = P(X ( (s) = ber (0) = n) 
_ P(X(s) =m, X(0) =n) 
P(X 0 )=n) 
2 5 PRS Sk 
= D P(X(0) =n) 
y P(X (s — )=k, X(0) =n) 


P(X(s — 5 =k, X(0) =n) 


= SOP(X(s—2) =k| X(0) =n) 


k=0 
| P(X(s) =m|X(s—¢2) =k, X(0) =n) 
= _ ~e)=k|X(0)=7n) 
k=0 


x P(X(s) =m | X(s —2) =k) 


= a ESE )Pk, m(€), 


k=0 
where the second-to-last equality is the Markov property. So, by (2.19), 


Pn.mlS) = Pam(s— E)Pm,m( £) FE 5 Pnk(S — £) 4k, m + 0(¢) 
km 


= Dnm s—é) 1 = $` pmi(é) +e 5° prnels — E)4k.m + C ole) 
im kžm 


= Pam(s—e)|1—€ >> dmi] +E >> pals — © dun + O(€) 
ižm kým 


= Pu. (s aiti e)(1 + €Gm,m) TE 5 Pn.k(S a 2)Gkm + ole). 
km 
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After a simple rearrangement we get 





8) — S-E Ole 
Disk ) Baim ) Pnm(8 NE Emm A 5 Pnls er E)Gk,m + ole) 
kým E 
oO 
QIE 
= XO pnels ~ €)Qk,m + oe 


k=0 


So, letting € — 0 in the previous equation, we get, for all n,m € Zt and all real s > 0, 
co 
Pnm(S) = > Pnk($)dk,m 
k=0 


as desired. o 


Using a similar argument, one can condition on the distribution of X (e), i.e., move 
forward in time from the origin (see Section 2.5.3), and then arrive at the Kolmogorov 
forward equations [20, 118, 161]: 


P(s) = QP(s). (2.24) 


Recall that 


Equipped with this initial condition, we can solve the Kolmogorov equations (2.23) for the 
case of a finite state space to get, for all t > 0, 


P(t) = e&t, (2.25) 
where the matrix exponential 
1 1 
exp(Qt) = 1+Qt+ Tha + gr +e. (2.26) 


Note that the terms ¢* /k! are scalars and the terms Q* (including Q? = I) are all square 
matrices of the same dimensions. Indeed, clearly exp(Q0) = J and, for all t > 0, 


II 


d 1 
q PQH) Q+Q%+ [Qh +. 


II 


(I+ Qt + Je +---]Q 
exp(Qt)Q, 


where, in the second equality, we could have instead factored Q out to the left to obtain the 
forward equations. 
In summary, for all s,t € R* such that s < t, the distribution of X (t) is 


a(t) = a7(s)P(t-s) 
a (s 


a(s) exp(Q(t — s)). 
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1 


Figure 2.5 An example transition rate diagram. 


@ EXAMPLE 2.4 


In this example, we will give an overview of a simple way to compute the matrix 
exponential exp(Qt) for the case where Q has distinct real eigenvalues. Consider 
the transition rate matrix (TRM) 


aor at a 
Q = 1 -1 0l. (2.27) 
1 0 =i 


The corresponding transition rate diagram (TRD) is given in Figure 2.5. 
We first compute the eigenvalues and eigenvectors of Q. The eigenvalues are the 
roots of Q’s characteristic polynomial: 


det(z21- Q) = 2(2+1)(2+3). 


Thus, the eigenvalues of Q are 0, —1 and —3. Taking z € {0,—1.—3} and then 
solving 


Qr = zz (2.28) 
for x (to determine the right eigenvectors corresponding to the eigenvalues) gives: 
e (1 1 1)" is an eigenvector corresponding to eigenvalue 0, 
e (0 1 —1]' is an eigenvector corresponding to eigenvalue —1, and 
e (2 — 1 — 1]" is an eigenvector corresponding to eigenvalue —3. 


Combining these three statements in matrix form, as (2.28) for a single eigenvector, 
gives 
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Let the diagonal matrix of eigenvalues of Q be 


0 0 0 
A= |o -1 0l, 
0 0 -3 


and let the matrix whose rows are the eigenvectors of Q in the same positions as the 
corresponding eigenvalues are in A be 


1 0 2 
V = 1 1 -1 
1 —1 -1 


Thus, we arrive at a Jordan decomposition of the matrix Q [30]: 
Q = VAV™. 
Note that, for all integers k > 1, 


QF = VAEV! 


where 
0 0 0 
AF = |o (-1)* 0 
0 0 (=3)* 


Substituting into (2.26) gives 


exp(Qt) = Vexp(At)V7! 
1 0 0 
= V| 0 eř* 0 y=, 
0 0 e” 


Note that we could have developed this example using left eigenvectors instead of 
right; for example, the stationary distribution g” is the left eigenvector corresponding 
to eigenvalue 0; see (2.29) below. 


Ways to compute the matrix exponential exp(Qt) for TRMs Q with nondistinct or 
complex eigenvalues are discussed in Chapter 8 of [30] or Chapter IV of [20]. 


2.4.5 The balance equations for the stationary distribution 


Suppose there exists a distribution g on the state space X = Z* that satisfies 


oO. = Ø, (2.29) 


X ondnm = 0 forallme Zt, (2.30) 
n=0 
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so that g is a nonnegative left eigenvector of Q with eigenvalue zero. Clearly therefore, for 
all integers k > 0, 


Thus, by (2.25), 


for all t € R*. Now consider the Markov chain X described above and recall that z(t) is 
defined to be the distribution of X (t). Therefore, if z(0) = ø, then z(t) = ø for all real 
t > 0. This is why øg is called a stationary or invariant distribution of the Markov chain X 
with rate matrix Q. The Markov chain X itself is said to be stationary if 7(0) = a. 





E EXAMPLE 2.5 


For the example TRM of Equation (2.27), the invariant distribution is 


]. 


oT = | 


wl 


1 
3 


w= 


E EXAMPLE 2.6 


The generation of Asynchronous Transfer Mode (ATM) packets from a human speech 
source can be modeled by a two-state Markov chain. First let the talkspurt state be 
denoted by 1 and the silent state be denoted by 0. That is, our modeling assumption 
is that successive talkspurts and silent periods are independent and exponentially 
distributed. In steady state, the mean duration of a talkspurt is 352 ms and the mean 
duration of a silence period is 650 ms [53]. The mean number of packets generated 
per second is 22, i.e., 22 48-byte payloads, or about 8 kbits per second on average. 
Solving (2.29) for a two-state Markov chain gives the invariant distribution: 


41.0 40,1 
oo = ———— and gy = ———. 
90.1 + 41,0 go.1 + 41,0 
From the above information, we get 
ae and EE 
11,0 = 7.352 101 = 0.650 


Finally, the transmission rate r packets per second during a talkspurt is determined 
by (2.36): 


0- oo tr-o = 22. 
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2.4.6 Transience and recurrence 


We now consider the properties of Markov chains that have bearing on the issues of existence 
and uniqueness of stationary distributions, i.e., of solutions to the balance equations TQ = 
0 and 


Xo; = 1. (2.31) 


Note that, by the definition of its diagonal entries, the sum of the columns of a rate matrix 
Q is the zero vector. Thus, the equations ¢'Q = 07 are dependent, thereby requiring 
another equation (i.e., (2.31)) if a unique solution øg is to exist. That is, we replace one 
of the columns of Q, say the 7th, with a column al of whose entries are 1, resulting in 
the matrix Qi. We then attempt to solve o™Q; = eT, where e; is a column vector whose 
entries are all zero except that the ¿th entry is 1. Thus, we are interested i in conditions on 
the rate matrix Q that result in the invertibility (nonsingularity) of Q; (for all i) giving a 
unique 


a” = aQ. 
Obviously, 
ci > O forall? 


is also required for ø to be a PMF on the state space. 
First note that the quantity 


Veli) = [ce = i} de 


represents the total amount of time the stochastic process X visits state 7. A state 2 of a 
Markov chain is said to be recurrent if 


P(Vx (4) = co | X(0) =7) = 1, (2.32) 
i.e., the Markov chain will visit state z infinitely often with probability 1. On the other hand, 
if 

P(Vx (2) = 00 | X(0) = 72) = 0, (2.33) 


i.e., P(Vx (i) < 00) = 1 so that the Markov chain will visit state 2 only finitely often, then z 
is said to be a transient state. All states are recurrent in the example of Figure 2.4, whereas 
all states are transient for the Poisson process. If all of the states of a Markov chain are 
recurrent, then the Markov chain itself is said to be recurrent. 

Suppose that ¿ is a recurrent state. Let 7; > 0 be the time of the first transition back into 
state i by the Markov chain. The state z is said to be positive recurrent if 


E(7, | X(0)=%) < œ. (2.34) 
On the other hand, if the state 7 is recurrent and 


E(r;|X(0) =i) = o, 
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Figure 2.6 An irreducible chain. 


then it is said to be null recurrent; see Section 2.5.3 for an example. If all of the states of 
the (temporally homogeneous) Markov chain are positive recurrent, then the Markov chain 
itself is said to be positive recurrent. Examples will be given in Section 2.5. 

Consider the transition rate diagram associated with the rate matrix Q of a Markov chain 
X. A rate matrix Q is irreducible if there is a path from any state of the transition rate 
diagram to any other state of the diagram. In Figure 2.6, an irreducible transition rate 
diagram is depicted. 

In Figure 2.7, a transition rate diagram is depicted that does not have a path from state 
2 to state 0; therefore, the associated Markov chain is reducible in this example. The state 
space of a reducible Markov chain can be partitioned into one transient class (subset) and a 
number of recurrent (or "communicating") classes. If a Markov chain begins somewhere in 
the transient class, it will ultimately leave it if there are one or more recurrent classes. Once 
in a recurrent class, the Markov chain never leaves it (when a single state constitutes an 
entire recurrent class, it is sometimes called an absorbing state of the Markov chain). For 
the example of Figure 2.7, {0,5} is the transient class and {1,2} and {3, 4} are recurrent 
classes. Note that irreducibility is a property only of the transition rate diagram (i.e., whether 
the transition rates are zero or not); irreducibility is otherwise not dependent on the values 
of transition rates. Finally, note that if the Markov chain has a finite number of states, 
then all recurrent states are positive recurrent and the recurrent and transient states can be 
determined by the TRD’s structure, as in Figure 2.7. 


Theorem 2.4.2. If the continuous-time Markov chain X is irreducible and positive recur- 
rent, there exists a unique stationary distribution g. 


For a proof of this theorem see, for example, Chapter 3 of [161] or Chapter IV of [20]. 
Again, example stationary distributions will be computed in Section 2.5. 
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Figure 2.7 A reducible chain. 


2.4.7 Convergence in distribution to steady state 


In the following theorem, the associated Markov chain X(t) ~ z(t) is not necessarily 
stationary. 


Theorem 2.4.3. For any irreducible and positive recurrent TRM Q and any initial distri- 
bution z(0), 


lim 7 (t) = Jim, x*(0)exp(Qt) = a’. (2.35) 


t-400 z 


That is, the Markov chain will converge in distribution to its stationary g; see [20, 161]. 
For this reason, g is also known as the steady-state distribution of the Markov chain X 
with rate matrix Q; see Problem 2.6(a) at the end of this chapter. More generally, for any 
summable function g on Z* 


Eg(X(t)) = So mtg) > J ogli) (2.36) 
i=0 i=0 


as t — œ [20]. 


2.4.8 Time reversibility and the detailed balance equations 


Consider a Markov chain X with transition rate matrix Q and unique stationary distribution 
ga. Take the entire real line R to be the time domain. Also consider the stochastic process 
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that is X reversed in time, i.e., define 


Y(t) = X(-t) 
fort € R. 


Theorem 2.4.4. The time-reversed Markov chain of X, Y , is itself a Markov chain and, if 


X is stationary, the transition rate matrix of Y is R whose entry in the mth row and nth 
column is 
On 
Tmn = nm —, (2.37) 
Om 


where qn,m are the transition rates of X. 


Proof: First note that the balance equations (2.29) ensure that 


o0 


N Taia SG, (2.38) 


n=0 
so that R is indeed a transition rate matrix. 
Consider an arbitrary integer k > 1, arbitrary subsets A, B, Bı, ..., Bp of Z*, and 
arbitrary times t, $, $1, .... Sk E Rt such that t < $ < s1 < ++- < Sk, 1€., 
-t> -s> —8, >:::> — Sk- 
The transition probabilities for the reverse-time chain Y are 


P(Y(-t)€ A| Y(-s) € B, Y(—s1) € Bi, ..., Y(—sx) € Br) 
= P(X(t)e A| X(s)€ B, X(s1) € Bı, n X(sx) € Bk) 
P(X(t) € A, X(s) € B, X(s1) € By, ..., X (sx) € Bx) 
P(X(s) € B, X(si) € Ba, ..., X(sx) € Be) 
P(X (sx) € Bk | X(t) EA, ..., X(sk-1) € Br-1) 
P(X (sx) € Bk | X(s) € B, ..., X(sk-1) € Bk-1) 
. P(X(t) € A, ..., X(sk-1) € Bk-1) 
P(X (s) EB, ..., X(sk-1) € Bp-1) 
P(X (sx) € Bk | X(sk-1) € Bk-1) 
P(X (sx) € Bk | X(sk-1) € By-1) 
P(X(t) € A, ..., X(s~-1) € Bri) 
P(X (s) E By ie X(sk-1) E Br-i) 
P(X(t) € A, X(s) € B, X(s1) € By, ..., X(Se-1) E€ Br-1) 
P(X(s) € B, X(s1) € Bı, ..., X(sk-1) € Br-1) 
where the second-to-last equality is a result of the Markov property of X. We can repeat 
this argument k — 1 more times to get 
P(Y (—t) € A| Y(-s8) € B, Y(-s1) € Bi, ..., Y(—s~) € Br) 
_ P(X(t) € A, X(s) € B) 
7 P(X(s) € B) 
= P(X(t) € A| X(s) € B) 
= P(Y (-t) € A| Y(-s) € B). 
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So, we have just shown that Y is Markovian. 
We now want to find R in terms of Q and g. Fort < s (i.e., —s < —t), note that 


P(Y (—t) =n|Y(-s) =m) 
P(X (t) =n | X(s) =m) 








— P(X(t) =n, X(s) =m) | P(X(t) =n) 
P(X(s)=m)— ~P(X(#)=n) 
= 2 m P(X(t) =n) 
= P(X(s)=m|X(t)=n)x P(X(s) =m) 
Since X is stationary by assumption, this implies that 
Pran(-t-(-8)) = Pam(s- OS, 


where n Æ m and the left-hand side is the transition probability function for Y. By (2.22), 
differentiating this equation with respect tos —t = —t-— (—s) and then evaluating the 
result at s — t = 0 gives 
a 
Tmn = qnm — (2.39) 
o 


m 


as desired. o 


Given the TRD R, it is easy to show that the reverse-time chain Y (t) = X (—t) also has 
stationary distribution g. Clearly, this should be true since the fraction of of time that Y 
visits any given state would be the same as the forward-time chain X. 

A Markov chain X is said to be time reversible if 


Tmn = mn foralln £m. (2.40) 


Many of the Markov chains subsequently considered will be time reversible. As we will 
now see, there is a simplified set of balance equations to compute the stationary distribution 
for time reversible Markov chains. Equations (2.37) and (2.40) together give the detailed 
balance equations for a time-reversible Markov chain: 


OmQmn = OnIn,m forall n #m. (2.41) 


So, X is time reversible if the average rate at which transitions from state m to n occur 
in reverse time equals the average rate at which transitions from state n back to m occur 
forward in time. 

If a distribution g satisfies the detailed balance equations (2.41) for a rate matrix Q, 
clearly it also satisfies the balance equations (2.30) for the invariant distribution of Q. 
Verifying this is left as an exercise. Given an irreducible and positive recurrent rate matrix 
Q, if one finds a distribution g that satisfies (2.41), then the associated Markov chain is 
time reversible. That is, time reversibility is a property that holds if and only if the detailed 
balance equations (2.41) are satisfied. 

Note that all two-state Markov chains are time reversible since the single balance equation 
is also a detailed balance equation. 
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E EXAMPLE 2.7 


Clearly, the TRM of Equation (2.27) is time reversible since the invariant is uniform 
(a; = a; for all ¿ # j) and the TRM is symmetric (qi; = q;,; for all i # j). 


E EXAMPLE 2.8 


Consider the following TRM: 


Q = 1 -2 1 


Its invariant distribution is 


la 

e] 

I 

— 
Ble 
wb 
le 
2a 


so that 


aq = 6-1 # $ 15041. 


Other examples of TRMs that are not time reversible are easily constructed by 
considering positive recurrent states i # j (i.e., oi, 0; > 0) with q; = 0 and 
dji # 0. 


2.5 BIRTH-DEATH MARKOV CHAINS 


We now define an important class of Markov chains on © = Z* that are called birth-death 
processes [20, 118]. From state zero, a birth-death process can only make a transition to 
state 1; let the rate of this transition be go,1. From state 7 > 0, a birth-death process can only 
make a transition to state 2 — 1 (at rate q; ;-1) or state i+ 1 (at rate qi ¿4.1). The terminology 
comes from Markovian population models wherein X (t) is the number of living individuals 
at time t, a state change from 7 to i + 1 represents a birth, and a state change from i to ¿i — 1 
represents a death. 
To simplify notation, we define 


Ài = Gi,i41 fori > 0 
and 


Mi = qii- for? >1. 
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2.5.1 Birth-death processes with finite state space 


In this book, we will consider two kinds of birth-death processes. One kind has a finite 
state space 


Z= Zt = {0,1,2,....K} 
and transition rates A; > Oforalli € {0,1,2,...,K —1}and yp; > Oforallz € {1,2,..., K} 


but Ax = 0. So, the finite birth-death process has an (K + 1) x (K + 1) transition rate 
matrix 





T —Ao Xo 0 0 0 met 0 
Hi =m A 0 0 ce 9 
0 H2 —p2—-A2 A2 0 an £ 
gal l l l i |. 242 
0 0 PE 0 UK-1 —PK-1) — AK =1 AK~1 
| 0 0 ees 0 0 HK LK 


Note that this rate matrix is irreducible. The finiteness of the state space implies that 
the birth-death process is also positive recurrent. We will now compute the stationary 
distribution g which is a vector of size K + 1. Our objective is to solve 


aQ = 0, 


which is a compact representation for the following system of K + 1 balance equations: 


— X00 +o = 0, 
N-10j-1 — (hi + Aici + Hi+1fi+1 = 0 for0<i< K, 
ÀK-10K-1 — KIK = Q. 


The solution to these equations? is given by 


i 
dj-1 
% = oo] 
J 


jar ” 





for0 <i < K and apo is chosen as a normalizing term (to satisfy (2.31) ), 


sal 





a = (uE) 


4 
aa pn 


It is left as a simple exercise to determine whether this Markov chain is time reversible and 
detailed balance holds. 


?For a systematic method to solve balance equations (which are difference equations), see the material on Z- 
transforms in [164] or the material on moment generating functions in [226]. 
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Figure 2.8 The transition rate diagram of an example finite birth-death process. 


E EXAMPLE 2.9 
We consider the example where 
A, =A and pisi pu 


for some positive constants A and yz. The transition rate diagram is depicted in Figure 
2.8. Define the constant 


61 


A 
P = =. 
H 
In this case 
P 
Oi = Tozy forl <i<K (2.43) 


and 


-1 
_ (ve 
Jg = vil. > 
n=0 ` 


Thus, the stationary distribution in this example is a truncated Poisson distribution. 


2.5.2 Birth-death processes with infinite state space 


The other kind of birth-death process considered in this book has an infinite state space ZT 
with transition rates A; > O for all ¿ > 0 and jz; > 0 for alli > 1. The transition rate 


diagram is depicted in Figure 2.9. In this case, the balance equations are 
—Aod0 +}101 = O 
and, for 2 > 0, 


N-10i—1 — (Hi + Ài)di + Hitsi = 0. 
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À À À À 
H H H H 
Figure 2.9 The transition rate diagram of an example infinite birth-death process. 


As for the finite case, the infinite birth-death process is irreducible. 
Assuming for the moment that it is positive recurrent as well, we can solve the balance 
equations to get 





io 
a= oJ 
for0 < i < K. Choosing go in order to satisfy (2.31), we get 


co i =l 
én = (S=) | 


i=1 n=1 





The condition for positive recurrence is that 


Cod | 
SI < x 


i=1 n=1 





because go > 0 under this condition and, therefore, g is a well-defined distribution. 


E EXAMPLE 2.10 


We now consider the example where A; = A and u; = u for all i and positive constants 
A and u. Again define the constant 


À 
p = -. 
u 
In the case where p < 1, 


v5 —1 
o = b> “) ’ 
i=0 


Note that 
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which converges (to 1 — p) if and only if p < 1; this is the condition for positive 
recurrence in this example. Thus, the stationary distribution when p < 1 is geometric 
with parameter p. The issue of recurrence for this birth-death Markov chain is further 
explored in the next section. 


2.5.3 Applications of the forward equations 


Consider an interval J = {i,i+1,...,.7 — 1,7} C Z of states, where i < j. Suppose a 
birth-death Markov chain X makes a transition into the interval at state i at time t, i.e., 
X(t) =iand X(t-) =i-1¢ J. Let Zy be the first time that X makes a transition to 
state k after time t, i.e., 


Ze = inf{s >t| X(s) =k}. 


Note that Z; = t by the definition of t above. Also, by the assumed temporal homogeneity, 
the distribution of Z; — t does not depend on t; so we take t = 0 to simplify notation in the 
following. Assume that the birth-death process is such that there is a positive probability 
that it will exit the interval J at either end. In this section, we will show how the Kolmogorov 
equations can be used to compute the probability that the Markov chain exits the interval J 
ati. Fork =i—1,1,...,9,7 + 1, define 


glk) = P(Zi-1ı < Zj41 | X(0) =k). 


So, we are interested in computing the quantities g(z) or g(j). 
First note that g(t — 1) = 1 and g(j + 1) = 0. Now consider a positive real number 
€ < 1 and note that, by a forward-conditioning argument, fori < k < j, 


g(k) 


So P(Zi-1 < Zj41 | X (0) =k, X (e) = m) 


x P(X(e) =m | X(0) = k) 
= J P(Zi-1 < Zj41 | X(€) = m)P(X(e) = m | X(0) =k) 


m 


|l 


g(k)(1 + Eqk,k) + x g(M)qk,mE + O(€), 
mk 


where the second equality above is just the Markov property itself. Recall that 
exp(eQ) =1+¢«Q+o0(e)U, 
where U is a square matrix in which every entry is 1. Also recall that 
dkk = 5 dkym = —4k,k~-1 — Gk,k4+1- 
m#k 
Therefore, 


k+1 
XO kam = 0 fori<k <j. (2.45) 


m=k—1 
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Note that (2.45) constitutes a set of j — i + 1 equations in as many unknowns (g(k) for 
i < k <j) with boundary conditions g(i — 1) = 1 and g(j + 1) = 0. The unique solution 
of these equations can be found by, for example, the systematic method of Z-transforms 
[164, 226]. In particular, the desired quantity g(7) can be found. 

Consider the example where q > O such that, for all k € J, 


dkk+1 = @ = dkk-1 
for some constant g > 0. In this case, the solution of (2.45) is 
g(k) = Ak +B 


for all k € J and for some constants A and B. These constants can be determined from the 
boundary conditions, i.e., 


gii-1)=AG—-1)+B, 
g(G +1) = AG +1) +B. 
Therefore, A = —1/(j — i + 2), B = (j +1)/(j — i + 2), and 


0 


j-k+1 
k) = =, 
g(k) as, 


Now consider the problem of computing the mean time until the return to a state 7 for 
the birth-death Markov chain with qk,k+1 = A and qx.4-1 = p. Considering that there 
are only finitely many states less than a given state 7, state 2 is positive recurrent only if 
hili + 1) < œ, where 


hij) = E(4; | X(0) = j). 


By using a forward-equation argument similar to that used to derive (2.45), we can obtain 


1 qjj+1 f 
hil) = —— + — h(j +1 
i( Qjj+1 gjj-  Qjj+1 + 95-1 f ) 


ihizi hi(j — 1) 














Qj.j+1 T fj, j—1 
1 À u 
= H hig +1) 4 hgj =i 2.4 
ee hea (+1) rra N ) (2.46) 
forall 7 > 7, with 
h(i) = 0 


by definition. One can see that this equation must be true intuitively by noticing that the 
first term on the right-hand side is the mean visiting time of state j and that the coefficient 
of h;(j £1) is the probability of transitioning from j to j + 1 in one step. If we define 


mG) = hiG)-hiG+1), 
then (2.46) becomes 


; 1 qj, j—1 : 
mj) = + 22 nlj- 1) 
dj.j+1 Qj.g+1 





1 1i 
ae a pu = 1), 
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Iterating, we get 


1 1\ 
mg) = ` ETE ani J 2) 
(oo? a. A 
= > = + ait). 
X 26 TG (i) 


Multiplying through by p’~* and then rewriting in terms of h; gives 


ae Lo 
phi) hil) = -hith 6%, (2.47) 
À k=1 
where we note that n; (i) = hili) — hili +1) = — hili +1). Now consider this equation 


as j —> oo. First note that the difference h;(j) — kilj +1) — 0. Now if p = 1, clearly 
(2.47) requires that h;(i + 1) = œ since the summation on the right-hand side is tending 
to infinity, i.e., state ¿į and, by the same argument, all other states are not positive recurrent. 
If p < 1, the summation on the right-hand side converges and the left-hand side tends to 
zero as j — œ so that 


1 p 


See ee, 
0 (i+ +3 Ip 


1 


hili +1) = p- 


A more general statement along these lines for birth-death Markov chains is given at the 
end of Section 4.7 of [118]. 


2.6 MODELING TIME-SERIES DATA USING A MARKOV CHAIN 


Consider a single sample path Xu (t), t € [0, T], of a stationary process, where T > 1. 
We may be interested in estimating its marginal mean, 


u = EX(t), 


by 


1 3 
7 I X(t at: (2.48) 


If this quantity converges to the mean as T — œ (for almost all sample paths X,,) then X 
is said to be ergodic in the mean. If the stationary distribution of X, ø, can be similarly 
approximated because 


F 
E ee f Enri; (2.49) 
T Jo 


T—=œæ 
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then X is said to be ergodic in distribution. In practice, estimates like (2.48) and (2.49) are 
sometimes used even when the process X is known not to be stationary assuming that the 
transient portion of the sample path will be negligible, i.e., the duration of the entire sample 
path T is much larger than the transient portion [85]. 

Given data (either from a physical system or a simulation) constituting a single sample 
path of a stochastic process, we would like to infer attributes of the probability “law” that 
governs it. We can perform such computations for processes X that are ergodic in some 
sense [207]. 

So far in this chapter, we have assumed that the TRM Q of a Markov chain is known. 
In this section, we describe a simple way to estimate transition rates from data. More 
specifically, the objective of this section is to obtain the most likely TRM Q given one or 
more measured sample paths (time series) of the physical process to be modeled, i.e., obtain 
the TRM for which the given sample paths are most likely (or "typical" [129]). 

We first assume that the states themselves are readily discernible from the data. Quanti- 
zation of the physical states would be required to obtain a discrete state space if the physical 
state space is uncountable. Even if the physical state space is already discrete, it may be 
further simplified by quantization (state aggregation). In the following, we assume that 
the physical process is positive recurrent and irreducible (leading to a unique invariant 
distribution). 

Given a space of N defined states, one can glean the following information from sample- 
path data, Xu: 


o the total time duration of the sample path, 7’, 


e the total time spent in state i, 7;, for each element 7 of the defined state space, i.e., 


T 
Ti = J 1{X (t) = i} dt, 
0 


e the total number of jumps taken out of state i, Ji, and 
e the total number of jumps out of state 7 to state j, Ji j. 


Clearly, 
T = a, Ti 


and, for all states 2, 


From this information, we can derive: 


e the sample occupation time for each state 7, 


e the sample probability of transiting to state j from 2, 


Jij 


Ji 


C 
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From this derived information, we can directly estimate the “most likely" transition rates 


of the process: 
Gj = Tig(-qi) foralli Æ j. (2.50) 


This leaves us with the V unknowns q;,; for 1 < i < N. We would like to use the N 
quantities g; to determine the residual N unknowns q;,;, but in order to do, so we need to 
assume that the physical process is stationary. If the physical process is indeed stationary, 
we can identify g as approximately equal to the stationary distribution of the Markov chain 
and therefore the balance equations hold: 


oQ = 0. 


Given that the substitution (2.50) is used for all ¿ 4 j in the balance equations, the result is 
only N — 1 linearly independent equations in N unknowns q;,; (recall that the columns of 
Q are dependent because they sum to zero). We note that the derived information q; and 
rij are all "relative" in nature. So, to obtain a final independent equation in the —g;,; so 
as ta specify them completely, we consider the total "speed" of the Markov chain, i.e., the 
aggregate mean rate of jumps: 


X oi(-4:.) = ZE Ji. 


E EXAMPLE 2.11 


Consider the following numerical example of process with N = 3 states with sample 
path data leading to the following information. The time-duration and occupation 
times were observed to be: 





T. To TA T3 


100 20 50 30 








The tota] number of transitions out of each state were observed to be: 


Jo A Je 
10 40 30 


The specific transition counts J; į; were observed to be: 





from\to| 0 1 2 


0 — 5 5 
1 10 — 30 
2 20 10 — 
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So, finding q;,; for all i # j as above gives 
71,1 - pa -Žž 
Q = -q2 42,2 -Boz 


20 10 
— 303,3 — 39 93,3 43,3 


Now the q;,; can be solved from the first two (independent) balance equations, 


20 50 , 10 30 , 20 = 
{0091.1 — i00 ° 10422 — T0 9093.3 = O, 
20. 5 50 30, 10 z 
— Too ` 1041.1 + 79092,2 — T00 ° 3093,3 = Q, 
and the total speed equation, 
20 50 30 1 
700 (— 41,1) + 395(—42,2) + yog(-93,3) = (10 + 40 + 30). 
The resulting solution is 
= AlT2 — _ 128 — _ 56 
Ni = ss, 92,2 = —g75, and 93,3 = —z. 


As mentioned previously, the state space itself can be determined from the data. In 
practice, a clustering algorithm could be used to decide or simplify the state space. Suppose 
that a histogram of transmission rates (in, say, bytes per second) of a stationary and ergodic 
packet flow has been empirically obtained. The individual samples of the transmission rates 
could, for example, be taken over a small sliding time window. The ultimate objective is to 
model the transmission rate process as a Markov chain. An example histogram A is given 
in Figure 2.10, i.e., A is a PMF so that 


Xala) = al, 


where the sum over the rate samples x was taken at a fine resolution (hence A is shown 
smooth in Figure 2.10). 

To simplify the Markov model, we wish to reduce the size of the state space to some 
given number n, i.e., n is small compared to the sample space of the histogram h. That is, 
given n, we want to define n — 1 thresholds 8; such that 


0 <A, <2- <n-1 < mw. 
The zth state of the Markov chain could then be given by 


T = 5 oe) 


6;-1<2<8; E 





where ĝo = 0, @,, = 00, and the (marginal) probability that the Markov chain is in state x7 


will be 
h = >> ho, 
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h 











Figure 2.10 Histogram of transmission rate process with the clustering thresholds 8. 


i.e., the TRM of the Markov chain will be chosen so that its invariant distribution is h”. For 
example, in Figure 2.11, we show a clustering with n = 6. 

There are several approaches to choosing the thresholds @ that, in turn, determine the 
state space 2* of the Markov chain. One could consider the variance of samples x within 
each cluster [106]. More precisely, note that the "total" variance of the raw samples is 

2 


Sof eS ene) | Ale) 


x £ 


n 


2 
=X jg- ih) KAA SO ha) (2.51) 
j=l 


i=1 i=l 6,-1<2<0; 


The proof of this identity is left as an exercise. The first term on its right-hand side is the 
intercluster variance while the second is the total intracluster variance. Again, given the 
number of clusters n, an objective of clustering could be to select the thresholds to minimize 
the intracluster variance, i.e., 


min) XO (w—a3)?h(2). 


9i—ıLr<ð; 
Techniques of clustering and Markov modeling are discussed briefly in [106] and in greater 
depth in [61, 175]. 
Given the state space, the data are distilled into a series of K state transition times, 
Tk > 0, where To = 0. See Figure 2.12, which depicts the quantizing/clustering operation 
on a sample path of trace data resulting in the sample path of the (quantized) Markov chain: 


{Th X (Tk) exo: 
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Figure 2.11 Histogram of transmission rate process with clustered states x”. 


2.7 SIMULATING A MARKOV CHAIN 


Simulation is a widely used tool in performance evaluation of communication networks. 
Care must be taken in the reporting of simulation results so that readers can precisely 
repeat the described experiments and so that the results are given to a prescribed degree 
of confidence [168]. In general, simulations are most effective in evaluating performance 
when they are combined with intuition obtained from the sorts of calculations that form 
the remainder of this book and from practical experience. Currently, popular simulation 
platforms for networking include OpNet (http://www.opnet.com), the open-source NS-2 
package (http://www.isi.edu/nsnam/ns), and the public Emulab (http://www.emulab.net). 
In this section, we describe a simple way to generate a sample path of a Markov chain by 
simulation. Suppose the state space is Z*, the initial distribution is 7(0), and the transition 
rate matrix is Q = [gn,m]. To find the initial state X (0), simply generate a uniform random 
variable Up and compute Fy '(Uo), where Fo is the CDF corresponding to 7(0), i.e., 


P = 0 ifr <0, 
o(z) = Dll x50) ife 2o, 


where |x] is the integer part of x € R; recall Problem 1.19. 
Now, the first transition of the Markov chain occurs at time 7), where T; is exponentially 
distributed with parameter —qx(9),x(0) > 0. We can generate another uniform (0, 1] 
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Figure 2.12 Sample path as empirically derived from quantized states. 


random variable V; (independent of Uo) and take 


1 
Tr = ee 0g (Vi). 
9x (0),X (0) 


As discussed in Chapter 2, the distribution z (T3) of X (T) is given by 


TmT) = @xX(0),m 
—4x(0)},X (0) 


= PXT) =m | X(0)) 
for m # X (0) and Txo) (T1) = 0. 


The second transition time occurs at time T2, where T2 —T; is anexponentially distributed 
random variable with parameter 


TAX (T1), X (Tı) 
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The distribution 7(T2) of X at time T% is given by 


ye (Ta) = dx (1),m 
4x (1),X(1) 
= P(X(Th) =m | X(N) 


form # X(1) and 7x(1)(T2) = 0. We can simply repeat this procedure to iteratively 
generate successive state transitions and intertransition times. 

Finally, the Markov chain itself could be a component in a more complex event-driven 
simulation. In this context, the (temporally ordered) event list would have a single entry for 
each Markov chain. The execution time of the entry would be the next transition time of the 
Markov chain. The entry is processed as described above: the state to which the Markov 
chain makes a transition and the intertransition time are computed, and the next entry for 
the Markov chain is inserted into the event list according to the latter. There are alternative 
simulation techniques for Markov chains including uniformization [35]. 


2.8 OVERVIEW OF DISCRETE-TIME MARKOV CHAINS 


In this section, we review concepts previously discussed but now for Markov processes in 
discrete time on countable state spaces, i.e., discrete-time Markov chains. We will highlight 
differences in the two contexts. Derivations that are completely analogous to the continuous- 
time case will be left as exercises. A follow-up section to this one, on discrete-time queues, 
concludes the following chapter. For further details on discrete-time Markov chains; see 
Chapter 3 of [118]. 

Recall that a stochastic process X is said be "discrete time" if its time domain is countable, 
e.g, {X(n)|n € D} for D = Zt or for D = Z, where, in discrete time, we will typically 
use n instead of t to denote time. Finite-dimensional distributions and the consistency and 
stationarity properties of a discrete-time stochastic process X are defined just as in (2.1), 
(2.2), and (2.3), respectively, but with the time values restricted to the discrete domain D, of 
course. The Markov property in discrete-time is also the same as that stated in continuous 
time in (2.8). In discrete time, the Markov property relies on the memoryless property of the 
(discrete) geometric distribution rather than on the (continuous) exponential distribution. 

If the random variables B(n) are IID Bernoulli distributed for, say, n € Zt, then B is 
said to be a Bernoulli process on Zt. Let q = P(Bn = 1). Thus, the duration of time 
B visits state 1 (respectively, state 0), is geometrically distributed with mean 1/{1 — q) 
(respectively mean 1/q). 

The analog to the Poisson counting process on Zt can be constructed: 


X(n) = $ Bim). 


m=0 


The marginal distribution of X follows a binomial distribution: for k € {0,1,2,...,n}, 


P(X(n-1)=k) = (a ~ grt, 


Both B and X are Markov chains. 
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Figure 2.13 The transition probability diagram for the Bernoulli process B. 


Figure 2.14 The transition probability diagram for the counting process .Y. 


The one-step transition probabilities of a discrete-time Markov chain Y are defined to 
be 


P(Y(n+1)=a|Y(n) =b). (2.52) 


where a. b are, of course, taken from the countable state space of Y. The one-step transition 
probabilities of a time-homogeneous Markov chain? can be graphically depicted in a transi- 
tion probability diagram (TPD). The transition probability diagrams of B and X are given 
in Figure 2.13 and 2.14, respectively. Note that the nodes are labeled with elements in the 
state space and the branches are labeled with the one-step transition probabilities. Graphicly 
unlike TRDs, TPDs may have "self-loops", e.g., P(X(n+ 1) =1|X(n) = 1) =q> 0. 

From one-step transition probabilities of a discrete-time, time-homogeneous Markov 
chain, one can construct its transition probability matrix (TPM), i.e., the entry in the ath 
column and bth row of the TPM for Y is (2.52). For example, the Bernoulli process B 
has state space {0. 1} and TPM 


Pg = ka ae (2.53) 
L=@ g 


That of the counting process X defined above is 


1-4 gq q 0 0 0 
0 l-@q q 0 0 


Pe = 0 0 l-q q 0 


That is, the quantity in the (2.52) Markov chain does not depend on time n for all states a. b. 
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Figure 2.15 The transition probability diagram on state space {0, 1, 2}. 


Another example of a discrete-time Markov chain with state space {0, 1,2} and TPM 


0.3 0.2 0.5 
Py = 05 0 0.5 (2.54) 
0.1 0.2 0.7 


is given in Figure 2.15. 
All TPMs P are stochastic matrices [99], i.e., they satisfy the following two properties: 


œ All entries are nonnegative and real (the entries are all probabilities). 
e The sum of the entries of any row is 1, i.e., P1 = 1 by the law of total probability. 


Given the TPM P and initial distribution 7(0) of the process Y (i.e., P(Y(0) = k) = 
Tk(0)), one can easily compute the other marginal distributions of Y. For example, the 
distribution of Y (1) is x7(1) = xT (0)P, i.e., 


m(1) = P(¥(1)=k) 
= S >P(Y(1)=k| ¥(0) = b)P(Y (0) = b) 


bES 


= 5 Po, To (0) 


bes 


for all & in the state space S of Y. By induction, we can compute the distribution of Y (n): 
a(n) = 2™(0)P”. (2.55) 


More general finite-dimensional distributions of a discrete-time Markov chain can be com- 
puted as Equation (2.16). Also, the quantity P” can be computed using the same approach 
as that described for the TRM (2.27). In this case, however, P will have an eigenvalue 
1 instead of zero corresponding to the eigenvector that is the invariant distribution, and 
powers A” of the diagonal matrix A are likewise simple to compute when all eigenvalues 
are distinct. 
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Note that a time-inhomogeneous discrete-time Markov chain will simply have time- 
dependent one-step transition probabilities, i.e., denote by P(n) the TPM made up of 
entries (2.52), so that (2.55) would become 

a(n) = x'(0)P(1)P(2)-+-P(n). (2.56) 
For a time-inhomogeneous Markov chain Y , the forward Kolmogorov equations in discrete- 
time can be obtained by conditioning on Y (1): 








(P(O.n)Jan = P(Y(n)=a]Y(0)= b) 
E P(Y (n) =a, Y (0) = 6, Y(1) = k) 
E 2, P(Y (0) =b) 
7 P(Y (n) = a, Y(0) = 6,Y(1) =k) P(Y(1) =k. Y (0) = b) 
D P(Y (1) = k Y(0) = db) P(Y (0) =b) 


il 


NC P(Y (n) = a| ¥(0) =, ¥(1) = k)P(Y (1) = k | Y (0) =b) 


Il 


k 
XC P(Y (n) =a|¥(1) = k)P(Y(1) = k | Y (0) = b). 
k 


where the second-to-last equality is the Markov property. The Kolmogorov forward equa- 
tions in matrix form are 


PO,n) = P(1L)P(1,7n). (2.57) 
Similarly, the backward Kolmogorov equations are generated by conditioning on Y (n — 1): 
P(0,n) = P(0,n-—1)P(n). (2.58) 
Note that both (2.57) and (2.58) are consistent with (2.56), i.e., 
P(0,n) = P(1)P(2)---P(n). 


Clearly, in the time-homogeneous special case, this simply reduces to P(0, n) = P”. 

For a time-homogeneous Markov chain, we can define an invariant or stationary dis- 
tribution of its TPM P as any distribution ø satisfying the balance equations in discrete 
time: 


og = go'P. (2.59) 


Clearly, by (2.55), if the initial distribution z(0) = g for a stationary distribution ø, then 
a(1) = g as well and, by induction, the marginal distribution of the Markov chain is g 
forever, i.e.,7(n) = g forall time n > 1 and the Markov chain is stationary. By inspection, 
the stationary distribution of the Bernoulli Markov chain with TPM (2.53) is 


De. 


The stationary distribution of the TPM (2.54) can be obtained numerically by solving 


a™(I-Py) = 0, 


dl = 1 
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Note that the first block of equations (three in this example) are equivalent to aT = gT Py 
and are linearly dependent. So, recall the idea to replace one of columns of I — Py, say 
column 3, with all 1’s (corresponding to 1 = aTl = do +0) +02) and replace 0 with 
[0 0 1] before attempting to solve these three unknowns from three equations. That is, we 
solve 


0.7 —0.2 1 
oo} -05 -1 1| = [00 1. 
—0.1 -02 1 


Note that the counting process X with binomially distributed marginal does not have an 
invariant distribution as it is transient. 

One can ask whether a given TPM has a unique stationary distribution. As in continu- 
ous time, individual states of a discrete-time Markov chain can be null recurrent, positive 
recurrent, or transient and we can call the Markov chain itself "positive recurrent" if all 
of its states are. Also, a discrete-time Markov chain can possess the irreducible property. 
Unlike continuous-time chains, all discrete-time chains also possess either a periodic or an 
aperiodic property through their TPDs (as with the irreducibility property). A state b of a 
time-homogeneous Markov chain Y is periodic if there is a time n > 1 such that: 


P(Y(m) =b| Y(0)=6) > 0 


if and only if m is a multiple of n, where n is the period of b. That is, given Y (0) = b, 
Y(m) = b is only possible when m = kn for some integer k. A Markov chain is said 
to be aperiodic if it has no periodic states; otherwise it is said to be periodic. The three 
examples of discrete-time Markov chains considered previously are all aperiodic. A Markov 
chain with TRD depicted in Figure 2.16 is periodic; note that n = 2 is the period of state 
2. One can solve for the invariant distribution of this Markov chain to get the unique 
a = [0,2 0.3 0.5], but the issue here is that the stationarity property of the Markov 
chain (2.3) does not hold because, e.g., if X (0) = 2, then X(n) = 2 almost surely (i.e., 
P(X (n) = 2 | X(0) = 2) = 1) forall even n and X (n) # 2 a.s. for all odd n. 
The proof of the following basic statement is given in [69]. 


Theorem 2.8.1. A discrete-time Markov chain has a unique stationary distribution if and 
only if it is irreducible, positive recurrent and aperiodic. 


Finally, we now discuss birth-death Markov chains with state space Z* because they 
will be used subsequently to study queues in discrete time. The counting process X defined 
above is a "pure birth" process. In Figure 2.17, the TRD of a birth-death process on a finite 
state space {0, 1, ..., K } is depicted (naturally assuming qx +p, < 1 forall k, where pp = 0 
and qx = 0). The balance equations (2.59) are 


(l-—qo)oo+pi91 = 90, 
Qn—19n-1 + (1 — qn —Pn)On + Pn410n+1 = On fordO<n< K, 
qkK-19K-1+(l1-pK)ok = ok. 


Subtracting the right-hand sides from the left-hand sides of these equations yields a set of 
equations just like those of Section 2.5.1 with terms à and yz respectively replaced by q and 
p. Thus, the solution of the stationary distribution here just as it was in the continuous-time 
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Figure 2.16 A periodic transition probability diagram. 
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Figure 2.17 Transition probability diagram for a birth-death process. 


setting: 


Oi = oJ] 2 


j=1 


for 0 < i < K and go is chosen as a normalizing term 


i=1 n=1 


ee (+o T 


Also, the example with qn = q and pn = np yields a truncated Poisson distribution (2.43) 
for o with p = q/p. 

One can also consider a discrete-time birth-death Markov chain on an infinite state space. 
Just as in the continuous time case, the process will be positive recurrent if and only if 


D a 


i=1 n=} 
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in which case o9 = (1 + R)7! and 


; 
Niger 

On = OQ . 
jai Pi 


The example where p, = p and qn = q also yields a geometric, invariant stationary 
distribution (2.44) with p = q/p < 1. Finally, by defining g and Z as in Section 2.5.3 for 
a discrete-time birth-death process, one can use a similar "forward" argument to show that, 
fori<k <j, 





g(k) = g(k+1)qk +g(k)(1— oe — pe) + 9(k — 1)pk, 


where the boundary conditions are g(i— 1) = 1 and g(7 +1) = 0. Note again the similarity 
between this equation and (2.45). 


2.9 MARTINGALES ADAPTED TO DISCRETE TIME MARKOV CHAINS 


Consider two discrete time stochastic processes Y and X neither of which are necessarily 
Markovian. The stochastic process Y is said to be a submartingale relative to the process 
X if 


E(Y (n) | X(n—1), X(n—2),...,.X(0)) 2 Y(n-1) 
almost surely for all time n. Note that 
E(¥(n)|X(n-1)) > ¥(n-1) 


if X is Markovian. The proof of the following basic convergence result is beyond the scope 
of this book; see Section 11.5 of [225]. We will apply it to a route selection problem in 
Chapter 6. 


Theorem 2.9.1. [f there is a constant A < œ such that sup,, E|Y (n)| < A, then there is 
a random variable Y such that 


P (lim ¥(n) =¥) =z 


NL OO 


ie., Y (n) > Y almost surely. 


Problems 


2.1 Verify that the theorems of Section 2.1 hold for geometrically distributed random 
variables. 


2.2 Extend Theorems 2.1.2 and 2.1.3 to involve more than two independent, exponentially 
distributed random variables. 


2.3 Show that exponentially distributed random variables are the only continuous, non- 
negative random variables with the memoryless property. 
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2.4 Show that a stationary process is wide-sense stationary but find an example where 
the converse does not hold. 


2.5 With regard to the statement of (2.18), consider two independent and exponentially 
(but not necessarily identically) distributed random variables X; and X2. For £ > 0, show 
that 


P(X, + X2 <e) = o(e). 


2.6 For a transition rate matrix Q with unique invariant distribution g: 


(a) Show that 


oT 

ot 

jim exp(tQ) = i 

gT 

Hint: Consider the statement of (2.35). 
(b) Verify this numerically for Example 2.4 and for 
2 -à à _ -A\-p 0 -1 

Q = ie A 0 iy i (2.60) 


where V is a matrix of right eigenvectors of Q: 
à 1 
AIET 


2.7 Show that the reducible Markov chain X with state space depicted in Figure 2.7 has 
infinitely many invariant distributions of the form 


0 0 
qo.1/(42,1 + %,2) 0 
91,2/ (42,1 +41,2) 0 
g. = i +a 
~ i 0 > | qa3/(03,4 + 94,3) 
0 q3,4/(93,4 + 94,3) 
0 0 


where œ; and ag are any positive real numbers satisfying a; + @2 = 1. Note that there are 
only two possible steady-state distributions to which the distribution of X (t) converges as 
t— ow: 


0 0 
q2,1/ (42.1 + %1,2) 0 
g1,2/(G2.1 + H,2) a 0 
0 q4,3/ (93,4 + 94,3) 
0 q3,4/ (93,4 + 94.3) 
0 0 
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2.8 Find the stationary distribution of the Markov chain with transition rate matrix given 
in (2.14). 


2.9 Verify Equation (2.38) and show that g is the invariant of the TRM R whose rates 
are given by (2.37). 


2.10 Are the Markov chain examples of Sections 2.5.1 and 2.5.2 time reversible? That 
is, check whether the detailed balance equations (2.41) hold. 


2.11 Ifa distribution ø satisfies the detailed balance equations (2.41) for a rate matrix Q, 
show that it also satisfies the balance equations (2.30). 


2.12 Consider a Poisson process X with rate À. Suppose that each transition time T; is 
independently considered and, with probability p, is deleted. So, the resulting transition 
times {7;} are a subset of the original {T;}. Show that the corresponding counting process 


Xx) = Suh <i 


i=l 


is a Poisson process with parameter (1 — p). Note: X is said to be a thinned version of 
the Poisson process X. 


2.13 Show that the examples of Figures 2.8 and 2.9 are time reversible and verify the 
detailed balance equations. 


2.14 Consider an interval J in R and suppose the number of transitions of a Poisson 

process N in J is given. In this context, show that the distribution of the transition times is 

equal to that of the ordered random vector of random variables that are IID uniform on J, 

i.e., show that 

“ dr; 

P(X; € (£a ti + dai), i=1,. n| NOU) =n) =n'[] r 
i= 


(2.61) 


where X; is the time of the first transition of N in J and 
T = IN. 


This is called the conditional uniformity property of a Poisson process. The Poisson process 
is the only counting process with this property, i.e., that satisfies (2.61) for any integer n 
and interval I. 


2.15 Describe how one could simulate the arrival times of a Poisson process over a fixed- 
length interval of time by using the conditional uniformity property. 


2.16 Consider a collection of IID random variables {Y;}%2_., independent of a Poisson 
process N on R. That is, 


N(B) = X HT: € B} 


7 


for (Borel) subsets B of R, where N(B) is a Poisson distributed random variable with 
parameter \|B| and |B| is the length (Lebesgue measure) of B. Let the interval (c,d) = 
Ry C R be the range of Yj, i.e., P(Y; € (c,d)) = 1, where -co < c < d < œ. Define 
the counting process 


N*(B) = SOUT +Y: €B}, 


t 
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i.e., shift the ith point T; of the Poisson process N by Y; to obtain NY. Show that NY is 
also a Poisson process with intensity À by showing that 


Eexp(sN* (I)) = exp(AlZ|(e*® — 1)). 
Hint: Consider an arbitrary interval J = (a,b), define I = (a — d,b —c), and note that 
|| = M| + |Ry|. Now, 
Eexp(sN*()) = S>E(exp(sN¥ (1) | N(Î) =n)P(N(L) = r) 
r=0 
and, by (2.61), the first term of the summand is 


exp (“x 1{X: +Y; € n) 


i=1 


E 








N(Î) = r = (Eexp(sl{U +Y € IþY, 


where U is uniformly distributed on Î and independent of Y ~ Y;. Finally, show that, 
irrespective of the distribution of Y, 
_ HI 
PU+Yeh=-—. 
[Z| 
Note: This problem represents a special case of a more general result; see [83] and Problem 
4.36 of [183]. 


2.17 Find the stationary distribution of the finite Engset birth-death Markov chain having 
K states and transition rates A; = (N —i+1)A and p; = ip for real positive constants A 
and yp and positive integers K and N. Show that when K = N, the stationary distribution 
is binomial. 


2.18 Show that the solution to (2.46) is 





for all j > i. Hint: Consider a solution of the form h;(j) = kj + a and solve for scalars «x 
and a. 


2.19 Prove (2.51). 
2.20 Compute the invariant distribution of Py given in (2.54). 
2.21 Prove the statement of (2.58). 


2.22 Suppose a transition rate matrix Q on a countable state space © is time reversible 
with invariant aø. Consider a real-valued function V on © and define the transition rate 
matrix 


Rey = Qz,ymin{1, exp(G(V(y) — V(2)))} 
for allz,y € Ð. 
(a) Show that R is also time reversible with invariant 


gx(B) = ea. red, (2.62) 
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where 


Z(8) = >_oyexp(—BV(y)) 


yE’ 
is the normalizing constant. 


(b) Show that as 8 — œc, the Gibbs distribution g(8) converges to a distribution that is 
concentrated on the subset of points at which V is minimized over ©. 


The quantity @ is interpreted as an inverse temperature, and when it decreases to zero over 
time in a sufficiently slow fashion, the resulting (time-inhomogeneous) Markov process, 
called simulated annealing, will converge (in probability) so as to minimize V on © [1, 90, 
97, 177], i.e., it will "track" the time-varying Gibbs distribution g(3(t)). 


2.23 Find the transition probabilities of the discrete time Markov chain whose state tran- 
sitions correspond to those of the continuous-time TRD in Figure 2.4, i.e., the "jump chain” 
with no self-transitions (Pn n = 0 forall n € {1,2,3}). 


CHAPTER 3 





INTRODUCTION TO QUEUEING THEORY 





This chapter introduces the theory of queues, specifically queues with a single input flow. 
Several topics of general scope are initially covered. A point will be made to apply these 
results in subsequent chapters. The second section is devoted to queues that can be described 
by an underlying Markov chain. The topic of the third section is the relationship between 
queueing delay and queue occupancy: Little’s result and the notion that Poisson arrivals 
see time averages. We conclude the chapter with a brief overview of discrete-time queues. 


3.1 ARRIVALS, DEPARTURES, AND QUEUE OCCUPANCY 


A queue or buffer is simply a waiting room with an identified arrival process and departure 
(completed "jobs") process. Work is performed on jobs by servers according to a service 
policy. In some applications considered in this book, what arrives to the queue will be 
packets of information. In other applications, the arrivals will represent calls attempting to 
be set-up in the network. In this chapter, we will simply call the arrivals jobs. 

Some jobs may be blocked from entering the queue (if the queue’s waiting room is full) 
or join the queue and be expelled from the queue before reaching the server. In this book, 
we will consider queues in which each job will either be blocked upon arrival or enter the 
queue and experience a queueing delay as it waits in the queue until it gets access to a server. 
A job’s queueing delay plus its service time is called its sojourn time, i.e., the time between 
the arrival of the job to the queue and its departure from the server. In this book, we will 
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consider queues that serve jobs in the order of their arrival; these queues are known as first 
come, first serve (FCFS) or first in, first out (FIFO). 

A counting process { A(0, t] | t € R+ } represents the number of jobs arriving at the queue 
over the interval of time (0, t]. Another counting process { D(0, t] | t € RY } represents the 
number of departures from the queue over (0, ¢]. Finally, the counting process { L(0, t] | t € 
Rt} represents the number of jobs blocked (lost) upon arrival over (0, t]. Let Q(t) be the 
number of jobs in the queueing system at time t; the quantity Q(t) represents the occupancy 
of the queue plus the number of jobs being served at time t; Q(t) includes the arrivals at 
t but not the departures at t. Clearly, a previously "arrived" job is either queued or has 
departed or has been blocked, i.e., 


A(0,t}+ Q(0) = Q(t) + D(O,t] + L(0,t]. 
If we take the origin of time to be —oo, we can simply write 


Q(t) = A(—co,t] — D(-oo. t] - L(-<«. t]. 





We will typically assume that the servers are nonidling (or “work conserving") in that 
they are busy whenever Q(t) > 0. Also, we assume a job’s service cannot be preempted 
by another job. Finally, we assume that all of the servers associated with a given queue 
work at the same, constant rate; thus, we can unambiguously define S; to be the service 
time required by the ith job. In addition, each job 7 will have the following two quantities 
associated with it: its arrival time to the queueing system T; (assumed to be a nondecreasing 
sequence in ¿) and, if the job is not lost, its departure (service completion) time from the 
server V;. 

At time ż, let Jo(t) C Z* be the index set of jobs in the queue and let Jg(t) C Z% be 
the index set of jobs that are being served. Thus, 


Q(t) = IH + Ist). 


Fori € Js(t), let R;(t) be the residual amount service time required by the ith job at time t. 
Clearly, 0 < R(t) < Sj, i.e., Ri(t) € (0, < Sj], for all i, t. For a queue with nonidling and 
nonpreemptive servers, we can define the total workload (or remaining work to be done) at 
time f, 


Wit) = $ G+ D> Rift). (3.1) 


iEJolt) i€ Js(t) 


For jobs 2 that are not lost, let V; be the departure time of the job from the server, i.e., from 
the queueing system. Clearly, V; — S; is the time at which the 7th job enters a server and, 
for all t andi € Js(t), 


R(t) = Vi-t; 


see Figure 3.1. Clearly, a job ¿ is in the queue but not in service (i.e., i € Jo(t)) if 

We will study lossy queues in greater detail later in this chapter. When considering 
a queueing system, it is useful to consider the arrival process A, queueing discipline, 
and departure process D separately. The arrival process A is parameterized above as 
{T;, Si}sez or z+. The queueing discipline determines how jobs are enqueued and in which 
order they are served (dequeued), i.e., the dynamics of queue Q and workload W processes. 


LOSSLESS QUEUES 85 


R;(t) 
aren = 
o Ñi 
poke aa i 
ph tH 
Ti V= t V; 


Figure 3.1 Queueing parameters associated with job ż that is not lost. 


The departure process, parameterized by {V;, S;} is determined by both the queueing dis- 
cipline and the arrival process. For a given arrival process and queueing discipline, we are 
typically interested in determining the "system" processes Q and W only in terms of the 
arrival parameters, i.e., not using the departure times V; as these may not be known a priori. 


3.2 LOSSLESS QUEUES 


In this section, we assume that the queue we have just introduced is lossless, i.e., L(—90, t] = 
0 for all t. Since, 


A(s.t] = XO HT: €(s,t]}, 


D(s,t] = DUM € (s,t}}, 


we get that 


Q(t) = A(-o0,t]- D(-co,t] = XO HT; <t< Vi}. (3.2) 


t 


Clearly, the total delay experienced by the ith job is 
Yea: 


i.e., the departure time minus the arrival time. The quantity V; — T; will also be called 
the sojourn time of the ith job. Again, this sojourn time consists of two components: the 
queueing delay, V; — S; -- T;, plus the service time, S;. 

In the following, expressions will be derived for quantities of interest such as the number 
of jobs in the queue, the workload, and job sojourn times. The objective is to express 
quantities of interest in terms of the job arrival times and service times alone. 
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Figure 3.2 A multiserver queue with no waiting room. 


3.2.1 No waiting room 


Suppose the queueing system consists only of the servers and no waiting room; see Figure 
3.2. Thus, if the job flow is demultiplexed (demux’ed) to one of K servers, the queueing 
system can only hold K jobs at any given time. Since the system is assumed lossless, 


Y= T+; 


for all jobs 7. Note that with infinitely many servers (K = oo), the system is always lossless 
and also that an infinite server queue with Poisson arrivals was the subject of Problem 2.16. 
Therefore, 


Qa) = PUN st<T +S} 


= X HT € (t- Sat} (3.3) 


For this situation, clearly Jo(t) = Ø and |Js(t)| = Q(t) for all t. This implies that, for 
all t, the workload satisfies 
2, Rife) 


W (t) 
i€ Ts (t) 


= X (Ti+ Si- t)HT; € ¢- Si t|} (3.4) 


Figure 3.3 depicts sample paths of Q and W. Note how the negative slope of the workload 
is proportional to the number of jobs queued. 
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Figure 3.3 Sample paths for an infinite-server queue. 


3.2.2 Single-server queue 


Suppose that the queue has a waiting room and only a single server, as depicted in Figure 
3.4. Clearly, if the waiting room was infinite in size, the queue would be lossless irrespective 
of the job arrival and service times. See Figure 3.5 for sample paths of the processes Q and 
W. Note that upon arrival of the ith job, Q increases by 1 and W increases by S;. The 
process Q is piecewise constant and, due to the action of the server, W(t) has zero time 
derivative if Q(t) = 0 (.e., W is constant) and otherwise has time derivative —1 for any t 
that is not a job arrival time. Upon departure of the ith job, Q decreases by 1. 


Theorem 3.2.1. For a work-conserving, single-server, lossless FIFO queue: 


Ve = mae Views Ti} + S: (3.5) 
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waiting room Server 


arriving jobs — O departing jobs 


Figure 3.4 A single-server queue. 


for all jobs i € Z*, where 


Proof: Consider the ith job arriving at the lossless queue. There are two cases. If T; > Vi-1, 
then job i — 1 has already departed the queue by time T;. Consequently, Q(T;—) = 0 (ie., 
the queue is empty just before time T;) and, when the 7th job joins the queue, it immediately 
enters the server. Therefore, it departs S; seconds after it arrives, i.e., V; = T; + Sj. 

On the other hand, if T; < V;_1, job 2 — 1 is present in the queue (and immediately ahead 
of the ith job) when the ith job joins the queue. Thus, the ith job will depart the queue S; 
seconds after job i — 1, i.e., V; = Vi-y + Sj. o 


Note that, by subtracting T; from both sides of (3.5), we get a statement involving the 
sojourn times V; — T; and the interarrival times T; — Ti—1: 


Vi-T; = max{(Vi-1 — Ti-1) — (Tı —Ti-1), 0} + S: (3.6) 
= max{Vj-1 —T;, 0} + S;, (3.7) 
where 
To = 0. 


An immediate consequence of the FIFO nature of a single-server queue is 

Vi = Ti+W(T). (3.8) 
Also note that 

max{V;_1, T;} 


is the time at which the ith job enters the server. 


3.2.3 Single server and constant service times 


A single-server queue is again considered. In this example, each job requires the same 
amount of service, i.e., for some constant c > 0, S; = 1/c for all i. So, the service rate of 
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Figure 3.5 Sample paths for a single-server queue. 


any server can be described as ¢ jobs per second. Further suppose that the (assumed lossless) 
queue has a waiting room. In this example, we have the following simple expression for 
the number of jobs in the system in terms of the workload: 


Q(t) = [eW(4)] (3.9) 


for all time t, where [zx] is the smallest integer greater than or equal to the real number zx. 
This equation is an immediate consequence of the fact that each job contributes c~! to the 
workload except for the job in service, which contributes only a fraction of c~}. That is, 
fe(Vi — t)] = 1 when the ith job is in the server at time t, i.e., when 


1 


1 
Vo= StV: e 0<V-t<-, 
c 


where we recall V; — t is the residual workload on the zth job. 
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Figure 3.6 Queue busy and idle periods. 











t 


We now prove an important expression for the workload which is not recursive in nature. 


Theorem 3.2.2. For awork-conserving, single-server, lossless, initially empty ( W (0) = 0) 
FIFO queue, 


1 
W(t) = u 74st] —- (t-s) (3.10) 


for all times t > 0, where the maximizing value of s is t if W(t) = 0, else the starting time 
of the busy period containing t. 


Proof: We first define a notion of a queue busy period as an interval of time [s, t) with 
s < t such that: 


(a) W(s—) = Q(s—) = 0, i.e., the system is empty just prior to time s, 
(b) W(r) >0 (and Q(r) > 0) for all time r € [s, t), and 
(c) W(t) = Q(t) = 0, i.e., the system is empty at time t. 


Queue busy periods are separated by idle periods, which are intervals of time over which 
W (and Q) are both always zero. So, the evolution of W is an alternating sequence of busy 
and idle periods; see Figure 3.6. Note that a busy period is, of course, initiated with the 
arrival of a job to an empty queue. 

Arbitrarily fix a time t somewhere in a queue busy period. Define b(t) as the starting time 
of the busy period containing time t, so that, in particular, b(t) < t and W(b(t—)) = 0. The 
total work that arrived over [b(t), t) is A[b(t), t]/c and the total service done over {b(t), t) 
was t — b(t). Since W (s) > 0 forall s € [b(t), t), 


W(t) = ~ Alb(t). t] — (t —b(t)). (3.11) 
Furthermore, for any s € [b(t), t), 


Wit) = W(s) + ŽAls, t] er 


IV 


= Als, ates, (3.12) 
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Now consider a time s < b(t). Since W(b(t)—) = 0, any arrivals over Ís. b(t)) have 
departed by time b(t); this implies that 


~ Als. b(t) — lt) ~ 5) wig (3.13) 
Therefore, 
“Als.]—(t-8) = =Als,0(6)) — (Olt) ~ s) + ŽAC), t] — (E — b) 
< TABE). - (t-a) 
= Wit). (3.14) 


Equation (3.11) and inequalities (3.12) and (3.14) prove the desired result for the case where 
W(t) > 0. 
The other case where Q(t) = 0 is similarly proved using the argument behind (3.13). O 


Note that, by Theorem 3.2.2 and (3.9), 


Q(t) = [ax Als —(t- sel (3.15) 


O<s 
Also, as a kind of converse to (3.9), when the ith job is in the server at time t (i.e., 


Is(t) = {i}), 


Wit) = * max{Q(t) -1,0} + Viet, (3.16) 


3.2.4 Single server and general service times 


Now consider a lossless FIFO single-server queue wherein the ith arriving job has service 
time S;. Let i(t} be the index of the last job arriving prior to time t, i.e., 


i(t) = max{j |T) <t} 
For this queue, it is left as an exercise to show that the workload is given by 


i(t) i 


Wit) = max a P|)! < (3.17) 


where (x) = max{z. 0}. 


3.3 A QUEUE DESCRIBED BY AN UNDERLYING MARKOV CHAIN 


A detailed survey of queues having underlying Markov chains is given in [226]. In this 
chapter, we will describe two of the simplest and most important examples of such queues. 
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The first is called the "M/M/1" queue, where the first "M" in this notation means that the job 
interarrival times are Memoryless; i.e., the job arrival process is a Poisson process which 
has exponential (memoryless) interarrival times Tp — Tn—1. The second "M" means that the 
job service times, Sn, are independent and identically distributed exponential (Memoryless) 
random variables. Also, the service times are independent of the arrival times. The "1" 
means that there is one work-conserving server. The queue is implicitly assumed to have 
an infinite capacity to hold jobs; indeed, "M/M/1" and "M/M/1/co" specify the same queue. 
So, the M/M/1 queue is lossless. When a general distribution is involved, the terms "G" 
or "GI" are used instead of "M"; "GI" denotes general and IID. So, an M/GI/1 queue has 
a Poisson job arrival process and IID job service times of some distribution that is not 
necessarily exponential. 

Let us now consider an M/M/1 queue with (Poisson) job arrivals of rate A jobs per 
second and (exponentially distributed) service times with mean 1/,: seconds. That is, the 
job interarrival times are independent and exponentially distributed with mean 1/2 seconds 
and, therefore, for all times s < t, A(s,t) is a Poisson distributed random variable with 
mean A(t — s). The mean arrival rate of work is À/ u and the service rate is one unit of work 
per second. 

On the other hand, the mean service rate can be described as u jobs per second. So, the 
queue (job) occupancy, Q, is a birth-death Markov process with infinite state space Z+. 
When the traffic intensity 


Q is the positive recurrent birth-death process with transition rate diagram given in Figure 
2.9. The stationary distribution of Q is geometric with parameter p; see Equation (2.44). 
The example of an M/M/K/K queue, i.e., a queue with A’ servers and no waiting room, is 
described in Section 3.5 below. The M/GI/oo queue was the subject of Problem 2.16. 

In the problems at the end of the chapter, we explore other queues with "underlying" 
Markov chains. In particular, from these underlying Markov chains we can derive infor- 
mation about the stationary mean workload of the queue. For example, one can show that 
the mean sojourn time of a job through an M/GI/1 queue is 


NE(S?) 
2(1 — AES)’ 


where, as above, S' is distributed as the service time of a job and A is the (Poisson) job 
arrival rate. This is a Pollaczek-Khintchine formula; see [219]. 


(3.18) 


3.4 STATIONARY QUEUES 


Recall that (3.6) is a statement about job sojourn times and Theorem 3.2.2 is a statement 
about queue occupancy. Also, in the previous section, we found the stationary distribution 
of the Q and W processes of the M/M/1 queue but not that of the sojourn times (delay), 
V — T, experienced by the jobs. 

Consider the ith job arriving at time T; to a FIFO single-server queue. Recall that the 
departure time of this job is given by 


Vn = Tr +W(T,). (3.19) 
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Suppose the queue is stationary. In this case, (3.19) implies the sojourn times of the jobs are 
identically distributed. Indeed, suppose we are interested in the distribution or just the mean 
of the job sojourn times. Regarding (3.19), one is tempted to identify the distribution of the 
sojourn times V — T with the stationary distribution of W; this gives the correct answer 
for the M/M/1 queue. In general, however, the distribution of W (Tn) (.e., the distribution 
of the W process viewed at a "typical" job arrival time Tn) is not equal to the stationary 
distribution of W (i.e., viewed at at typical time); see Section 3.4.3. 

In this section, we will explore the relationship between the stationary distribution of a 
queueing system (i.e., as viewed from a typical time) and the distribution of the queueing 
system at the arrival time of a typical job. We will motivate, state, and apply Little’s result 
and the rule that Poisson arrivals see time averages. 


3.4.1 Point processes and queues on R 


A stationary Poisson process on the whole real line R is defined by a countable collection 
of points {7;}22_.,, where the interarrival times 7; — 7;—; are IID exponential random 
variables. Alternatively, we can characterize a Poisson process on R by stipulating that the 
number of points in any interval of length ¢ is Poisson distributed with mean Aż and that 
the number of points in nonoverlapping intervals is independent; recall Problem 2.16 of 


Chapter 2. 


E EXAMPLE 3.1 


This last characterization naturally extends to a characterization of a Poisson point 
process on R” for all dimensions n > 1, i.e., a spatial Poisson process: If v( A) is the 
volume (or area) of A C R”, then the number of points in A is Poisson distributed 
with mean Av(A). 


Let A(s. t] be the number of jobs arriving at a work-conserving, lossless, FIFO queue 
over the interval of time (s, t], where s, t € R. Let c jobs per second be the constant service 
rate of the queue, i.e., each job requires 1/c units of work and the server can execute one 
unit of work per second. Assume that the point process {7;}?2_ of job arrival times is 
stationary and not necessarily Poisson. In particular, this means that the distribution of 
A(s. t] depends on s, t only throught — s, i.e., A(s.t) ~ A(s+r.t+r) forall s.t.r ER. 
Clearly, we have 


xX 


A(s.f] = 5 l{s <7; <t}. 


{=_e 


Note that some jobs have negative indices. 
For any time ¢ € R, the number of jobs in the queueing system at time t is given by 


Qi) = | max Alst] = et = s)| (3.20) 


—oo<s<t 
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Figure 3.7 Stationary version of a marked point process M. 


and the workload is given by 


1 
W(t) = max, z4(s,t] -— (t-s); (3.21) 


see Equations (3.9) and (3.10). Again, note that the "maximums" are taken over all real 
times s < t. 


3.4.2 Stationary and synchronous versions of a marked point process 


Consider again a stationary point process {7;}92_,, on R. Suppose the subscript labels are 
chosen so that To < 0 < 7. To each point 7; we associate a mark (random variable) y;. 


The ensemble 
M = {aiwies 


is called a stationary marked point process, see Figure 3.7. One can interpret that the 
marked point process is viewed at a typical time in Figure 3.7, where the typical time is the 
origin 0. That is, the process "began" infinitely long ago and, by time 0, has reached steady 
state. Alternatively, given the positions and marks on the real line, the origin of time 0 is 
selected "uniformly at random" (hence the term typical) and thereby fixes the indices of the 
marks. 

Now consider the marked point process as viewed from a typical arrival; see Figure 3.8. 
That is, an arrival is chosen uniformly at random and the origin of time is set to the arrival 
time of that mark. To this end, define the synchronous marked point process 


Mo = {Tr ti} 


where the subscripts of the point process T; are chosen so that To = 0. So, zo is a typical 
mark and 7; is the length of a typical interarrival time. As indicated above, we will interpret 
each point as an arrival time of a job to a queueing system. A mark associated with a job 
will, in some way, represent the "state" of the queueing system at the job’s arrival time. A 
translation of a marked point process to the left by a fixed amount of time t € R is denoted 
by 


Mo+t = {Ti — t, zi ra 


i=—00" 
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Figure 3.8 Synchronous version of a marked point process Mo. 


Since M is stationary, it is distributed as M + t, i.e., 


M+t ~ M, 
for all t € R. 
Two basic assumptions in this book will be that the intensity of the point process 
1 
A = = 
ET, 


satisfies 0 < À < oo and that the processes under consideration are ergodic [138]; see 
Section 1.11. In particular, ergodicity implies that the law of large numbers for M (and 
Mo) holds: For all times s € R, 








A(s, 
lim (s.t) Aà almost surely, 
tox t-s 
and for all times t € R, 
A(s, 
lim E = AÀ almost surely. 


s—=-x t-s 


Under these conditions, Loynes showed [9, 14] that there exists a stationary solution W to 
(3.21) with W (0) distributed as the one-dimensional (stationary) marginal of W. 


E EXAMPLE 3.2 


In this example, differences between M and Mo are demonstrated. Consider a point 
P p 
process on R whose interarrival times are discretely distributed as 


P(T, =5)=4 and P(T =10)=4. 


The distribution of 7, — To will now be computed. Consider a large interval of time 
H > 1 spanning N consecutive interarrivals. Recall that for the stationary process 
M one can interpret the origin of time as uniformly randomly selected. Thus, the 
probability of selecting the origin of time in an interarrival of length 10 is equal to 
the fraction of interarrivals of length 10 that cover H. That is, since H > 1, by the 
law of large numbers (Section 1.11), H ~ NET; = N(5-4 +10- 2), Therefore, 
N -5(1/4) 


1 
P(m-—T=5) = N(5(1/4) + 10(3/4)) ~a 
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3.4.3 Poisson arrivals see time averages 


The following is a general “conservation law" called Palm’s theorem [14, 219] that we state 
without proof. 


Theorem 3.4.1. 
Ty 
EG(M) = re f G(Mo + t) dt 
0 


for any real-valued function G. 
The following are immediate corollaries of Palm’s theorem. 


Lemma 3.4.1. For all v,w > 0, 


P(-7 >v n >w) = af P(T; >t) dt. (3.22) 
v+w 
Proof: Applying Palm’s theorem with 
G(M) = 1{-m>v, n >w} 

results in the following right-hand side: 

Tı Tı 

Ef GMo+t)dt = aE f iG Sach ats wid, 
0 0 


where we note that To = 0 < t < Tj inthe integrand implies that Tp -t = —t < 0 < Ti —t, 
i.e., the points indexed 0 and 1 of Mo + t are on different sides of the origin. Thus, 


Ti oO sS 
AE G(Mo + t)dt = af i 1{t>v, s—t>w}dt P(T; € (s,s + ds)) 
0 o Jo 


ll 


af / 1{t>v, s—t> w}P(T; € (8,8 + ds)) dt, 
0 t 


where the last equality is simply switching the order of integration (Fubini’s theorem). 
Since the integrand is 1 if an only if t > v ands > t+ w, 


Ty 


ME | G(My +t)dt 
0 


af P(T; € (5,5 + ds)) dt 
v t+w 


= af P(T, >t+w)dt 


af P(T > t) dt, 
utw 
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where the last equality is just a change of the variable of integration to té=t+w. o 


Lemma 3.4.2. T; is exponentially distributed with parameter À if and only if 


Pia >w) = PT >w). (3.23) 


Proof: Applying Palm’s theorem with 
G(M) = l{n>w} 


results in the following right-hand side: 


Tı Tı 
AE f G(Mo+tidt = AE i 1{T, — w > t} dt 
Jo Jo 


= af I 1{s— w >t}dt P(T) € (s.s + ds)) 
0 0 

= af f 1{s -w > HPT E (5.5 ES ds)) dt 
os D 

E af i, P(T, € (s.s + ds)) dt 
0 t+w 

E af P(T, >t + w)dt. 
0 


Now, since 


bapi f P(T, > t) de 
À 0 


(recall Problem 1.11) and since A C B implies that P(A) = P(A|B)P(B) (assuming 
P(B) > 0), then 


So PGi > t+w|T >t)P(T, > t)de 
fo P(T, > t)dt 








Ti 
re f G(Mo + t) dt 
JO 
Therefore, 
aA 
AE G(Mo +t)dt = P(T; > w) forall w > 0 
0 
if and only if 


P(T >t+w|T, >t) = P(T,>w) forallw > 0. 


i.e., Tı is memoryless. Since the exponential is the only memoryless continuous distribu- 
tion (Problem 2.3 of Chapter 2), the lemma is proved. 
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Figure 3.9 Synchronous workload of a single-server, lossless, nonidling queue, G(Mo). 


Now suppose that {7;}2_., is a Poisson process (with mean rate à), i.e., {Tj — 


Ti-1} -ə are IID exponential random variables each with mean 1/. Lemma (3.4.2) 
implies that 7, is exponentially distributed with parameter A. By Lemma 3.4.1, 


P(-7) >v,71>w) = ene re) 
=> P(-m>u|[n>w)P(n >w) = eAetr) 
=> P(-mn>v|n >w) = e™™ 


forall v, w > 0. Letting w — oo, we see that To and 7; are independent and —7» is also 
exponentially distributed with parameter A. Therefore, if we insert a point at the origin, 
the point process {7;}92_, remains a Poisson process! That is, the stationary Poisson 
point process {7;}92_.., with a point inserted at the origin is identical in distribution to the 
synchronous Poisson point process {T;}%_... 

Now consider the stationary, lossless, FIFO, single-server queue with Poisson arrivals 
depicted in Figure 3.10. Let 7; be the arrival time and y; be the service time, respectively, 


of the 7th job. Recall the statement of (3.17) concerning the stationary workload at time 0: 
+ 


0 
w(0) = max 2 yx) +7;| = GM). 
=j 
The implication of the previous paragraph is that 


where @ is distributed as the service time of a typical job.! Thus, 


<0 
aak 


0 
W(0) ~ max X` sk +T}, 
j 


! Equivalently, M ~ Mo\ {To = 0, zo}. i.e., remove the point at the origin of Mo. 
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Figure 3.10 Stationary workload of a single-server, lossless, nonidling queue, W (0) = G(M). 


where the right-hand side is the workload upon (just before) the arrival of a typical job. 
This is one instance of the rule called Poisson arrivals see time averages (PASTA). By the 
same argument above, PASTA can be applied to any stationary queue (or queueing system) 
wherein the "state" (workload, number of jobs, etc.) just prior to time 0 can be written as a 
function of {7;, y;};<o, i.e.. the queue is causal or nonanticipative. 


E EXAMPLE 3.3 


Consider the stationary M/M/1 queue with traffic intensity p = A/p < 1; recall the 
example of Equation (2.44). By PASTA, the distribution of the number of jobs in the 
queue just before the arrival time T of a typical job? is geometric with parameter p: 


P(Q(T—) =i) = (1—p)p' forie Z*. 


Note that Q(T) = Q(T-) +121. 

The sojourn time of an arriving job is the total workload upon its arrival in the 
queue. To find the distribution of the sojourn time of a typical job, W (T`). let {S, } be 
an IID sequence of random variables that are exponentially distributed with parameter 
u. Since the service times are IID and exponentially distributed, the total workload, 
W(T), is distributed as 


Š; | + R1{Q(T) > 0}. (3.25) 


2T = To = Oin the notation above. 
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where 


and R is distributed as the residual service time of the job in the server at time T. The 
memoryless property implies that R is exponentially distributed with parameter 4 as 
well, i.e., simply that 


Q(T-)+1 Q(T) 
wr) ~ $ & = 1S. 
j=l j=l 


Consequently, given Q(T—) = n, W(T) (the sojourn time of a typical job) has a 
gamma (Erlang) distribution with parameters n + 1 and x. 

The mean sojourn time w can now be obtained using an instance of Wald’s identity 
[226]: 


EW(T) = EQ(T)ES (3.26) 
p 1 
( =) H 
1 


where we used the formula for the mean of the geometrically distributed random 
variable computed in Section 1.4.2. 


Without Poisson arrivals, the implication of PASTA usually does not hold [15]; recall 
Example 3.2. 


3.4.4 Little’s result 


Campbell’s theorem (interpreted as a converse of Palm’s theorem 3.4.1) allows us to compute 
the distribution of M from that of Mo for times t € R: 


Theorem 3.4.2. For any nonnegative function f on R?, 


re | f(t,ao)dt = E >> firm). 


1=—00 


Proof: (Sketch [219].) Consider a small interval of time (t, t + dt) and the distribution of 
a typical mark, P(zo € (x,x + dax)). The quantity P(xo € (z,2 + dx)) would be equal 
to the fraction of points of the stationary process M in the interval (t, t + dt) whose marks 
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y are in the interval (x, z + dx). Note that the expected number of points in (t,t + dt) is 
A dt. The expected number of marks y € (x, x + dz) in this interval of time is 


E 5 1{y; € (x, £ + dz), t; € (t,t + dt)}, 


i=% 


where we note that the summand does not depend on t by stationarity. Therefore, by 
ergodicity, 


1 oo 
P(zo E€ (z,xz + dz)) = e y 1{y; € (x,x£ + dx), 7; € (t.t + dt)}. 


i=—00 


Multiplying this equation by f(t, x) and integrating both sides over R? gives the desired 
result. oO 


A rigorous treatment of Palm and Campbell’s theorems is given in [14]. 

Suppose that the queueing system is lossless and define the mark yx as the sojourn time? 
of the job arriving at time Tk, i.e., the departure time of the kth job is Tk + Yp (note that 
neither Palm nor Campbell’s theorem requires any kind of independence among the arrival 
times or marks of the point process). Also define 


flny) = 1{7<0,7+y>0} 


so that f(T. Yk) = 1 if and only if the kth job is in the queueing system at time 0, the 
typical time at which the stationary process M is viewed. Let w be distributed as the mean 
workload in the queue upon arrival of a typical job and let L be the mean number of jobs 
in the system in steady state. Thus, 


[oe] 
L = EY l{r<0,r+y>0} 


i=— 00 
and so, by Theorem 3.4.2, 


0 
L = af El{zo > —t} dt 


-o 


oO 

= af P(xo > s)ds 
0 

= AEzo, 


where the variable of integration was changed from t to s = —t. Finally, note that w = Exo. 
Therefore, we have arrived at Little’s result: 


Theorem 3.4.3. 
L = dw. 


35o, for a lossless, FIFO, single-server queue, yg = W (Tk), which is a function of {Ti }i<x and the associated 
service times. 
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To reiterate, Little’s result relates the average number of jobs in the stationary lossless 
queueing system (i.e., the average number of jobs viewed at a typical time 0) to the mean 
workload in the queue upon arrival of a typical job (i.e., the mean sojourn time of a job). 
Theorems 3.4.1, 3.4.2, and 3.4.3 are "conservation" laws of very general applicability. 


E EXAMPLE 3.4 


By (2.44) and the calculation of the mean of a geometrically distributed random 
variable in Section 1.4.2, the mean number of jobs in the stationary M/M/1 queue is 


where we recall that p = \/ 1 < 1 is the traffic intensity. 
By Little’s result, the mean workload in the M/M/1 queue upon arrival of a typical 
job (i.e., the mean sojourn time of a job) is 


which agrees with (3.27). 


Now consider again a lossless, FIFO, single-server queue Q with mean interarrival time 
of jobs 1/\ and mean job service time 1/4 < 1/4, i.e., mean job arrival rate A and mean 
job service rate u > A. Suppose the queue and arrival process are stationary at time zero. 
The following result identifies the traffic intensity \/ with the fraction of time that the 
stationary queue is busy. 


Theorem 3.4.4. 


P(Q(0)=0) = 1-2, (3.28) 


Proof: Consider the server separately from the waiting room (as we have not done so far in 
this book). Assuming that the mean departure rate of the waiting room is À too (see below), 
Little’s result implies that the mean number of jobs in the server is L = A/p. Finally, 
since the number of jobs in the server is Bernoulli distributed (with parameter L), the mean 
corresponds to the probability that the server is occupied (has one job) in steady state. O 


We state the following result for future reference. 
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Figure 3.11 Departure rate of a stable (A < 4), lossless, nonidling single-server queue. 


Theorem 3.4.5. For a stable lossless queue, the mean (net) arrival rate equals the mean 
departure rate in steady state, i.e., 








lim 
too t t— o0 t 


Proof: The queue is stable implies that Q(t)/t — 0 almost surely as t — œœ. Recall 
that 


Q(0)+ A(0,t] = Q(t) + D[0,t), 


where A(0,t) and D(0,t) are the cumulative arrivals and departures over (0, t), respec- 
tively. Dividing this equation by t and letting t — oo gives the desired result. on 


See Figure 3.11, indicating the departure rate corresponding to a queue’s sample path. 
Note that the mean departure rate is 


w-P(Q>0)+0-P(Q=0) = u-p 


which is consistent with the statements of the previous two theorems. 
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3.5 ERLANG’S BLOCKING FORMULA FOR THE M/M/K/K QUEUE 


In this section, we study another important example whose queue occupancy can be de- 
scribed by a finite birth-death Markov chain. Consider a queue with Poisson arrivals, ITD 
exponential service times, K servers, and no waiting room, i.e., in the notation of Section 
3.3, an M/M/K/K queue.* As in Section 3.3, let A be the rate of the Poisson job arrivals 
and let 1/4 be the mean service time of a job. 

Suppose that there are n jobs in the system at time t, i.e., Q(t) = n. We can adapt the 
argument at the end of Section 3.3 to show Q is a birth-death Markov chain and finds its 
transition rate diagram. Indeed, suppose Q(t) = n > 0 and suppose that the past evolution 
of Q is known (i.e., {Q(s) | s < t} is given). By the memoryless property of the exponential 
distribution, the residual service times of the n jobs are exponentially distributed random 
variables with mean 1/4. Therefore, by Theorem 2.1.2, Q makes a transition to state n — 1 
at rate nu,ie.,forO<n< K 


dn.n-1 = Mp. 


Now suppose Q(t) = n < K. Again by the memoryless property, the residual interarrival 
time is exponential with mean 1/\. Therefore, Q makes a transition to state n + 1 at rate 
A, ie  fr0 < n< K 


Qnyntli = À. 


Thus, this is the Markov chain described in the example of Section 2.5.1 with transition rate 
diagram give in Figure 2.8. The stationary distribution of Q is the truncated Poisson given 
in Equation (2.43): 


ø” 
oi = o forl<i< K, 
i! 


K ;\7t 
09 = pr : 

5 u 

i=0 


Now considera stationary M/M/ K/K queue. Suppose we are interested in the probability 
that an arriving job is blocked (dropped) because, upon its arrival, the system is full, i.e., 
every server is occupied. More formally, we want to find P(Q(T,,—) = K), where we 
recall that T» is the arrival time of the nth job. Since the arrivals are Poisson, we can invoke 
PASTA to get 


and 


K 
P(QIn-)=K) = ox = oo a (3.29) 
which is called Erlang’s formula. Note that this result depends on the distribution of the 
job service times S; only through the mean 1/1, which not surprising since an exponential 
service time has only one parameter. For more general service time distributions, it can be 
shown that (3.29) still holds. Therefore, given the mean service time 1/1, Erlang’s result 
is said to be otherwise "insensitive" to the service time distribution [224, 226]. 


4Since the capacity to hold jobs equals the number of servers, there is no waiting room (each server holds one 
job). 
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Finally note that, by Little’s theorem, the mean number of busy servers in steady state is 


L 


MI =x) 
= p(1 gi ok), 


where A(1— øx ) is the "net" arrival rate of jobs that successfully join the queue (by PASTA). 
One can easily check that 


L = EQ. 


3.6 OVERVIEW OF DISCRETE-TIME QUEUES 


Consider now a single-server queue in discrete time n € Z* or n € Z. Suppose that the 
server works at a normalized rate of c jobs per unit time and that a,, is the amount of work 
that arrives at time n. If we assume that, in a given unit of time, service on the queue is 
performed prior to accounting for arrivals (in that same unit of time), then the work to be 
done at time n is 


W(n) = (W(n—-1)—c)* +an, (3.30) 
where, again, 
(€)* = max{0,é}. 
Alternatively, if the arrivals are counted before service in a time slot, 
W(n) = (W(n-1)-—c+an)*: (3.31) 


these dynamics are sometimes called “cut-through" because arrivals to empty queues can 
depart immediately incurring no queueing delay. 
For the dynamics (3.31), the following expression holds: 


W(n) = max [A(m, n] — e(n —m)], (3.32) 
where 


A(m,n} = am+1 +8am+2 + + an. 


For the dynamics (3.30), 


W(n) = an+ max [A(m,n) — e(n- m), (3.33) 
where 
A(m,n) = am+1 +4am42 +` + an-1. 


One can verify that (3.31) implies (3.32), as (3.30) implies (3.33), by a straightforward 
induction argument. Conversely, (3.32) and (3.33) can be obtained just as in the continuous- 
time examples by showing that the time m that achieves the maximum is the start time of 
the busy period of the queue that contains time n. 
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E EXAMPLE 3.5 


Consider a layer 2 FIFO queue in a communication network receiving fixed-length 
frames of bytes with a, frames arriving at time n. If the nonidling server works at 
rate c = 1 frame per unit time, the number of frames in the queue Q(n) = W(n). 
Discrete-time queues with service rate less than 1 frame per unit time are described 
in Chapter 2 of [126]. 


Now consider a FIFO queue with a single nonidling server and infinite waiting room 
in discrete time. Suppose that the job interarrival times are IID geometrically distributed 
with mean 1/g. The service times of the jobs are also IID geometric with mean 1 /p, where 
p = q/p < 1. So, the number of jobs in the queue Q is a birth-death Markov chain as 
depicted in Figure 2.17, i.e., a discrete-time M/M/1 queue. From the invariant distribution 
a, the mean number of jobs in the queue is 


oO 


L = Si, = otis: (3.34) 


k=0 


Thus, by Little’s formula in discrete time, the mean sojourn time is 1/(p — q). 


E EXAMPLE 3.6 


Consider a TDM wireless downlink of a base station (BS) fed by a single layer 2 FIFO 
transmission queue of fixed-length frames. Assume that the base station serves a group 
of N end hosts and that the transmission error events from the BS to the end hosts are 
UD and occur with probability 1 — pin any given time slot. A frame is transmitted by 
the BS in consecutive time slots until it is received error free. Also assume that the 
frame arrival process to the downlink transmission queue is a memoryless counting 
process (Figure 2.14), independent of transmission error events, with "mean rate" 
q < p frames per time slot. For any arbitrarily selected frame, the probability that a 
frame belongs to flow i is q;/q, where 


N 
Se 
i=l 


We want to compute the mean delay of type 7 frames through this queue. 
First note that, due to transmission errors, the service time of a frame is effectively 
geometrically distributed with mean 


Ip + 2p(1—p)+3p(1—p)?+--- = (3.35) 


Ble 


So, for the queue to be "stable," we require that 


p = < 1. 


S PA 
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By PASTA, the mean sojourn time w; of type i frames is equal to that of the aggregate 
flow, w, i.e., w; = w for all flows 7. By Little’s formula, the mean number of type 2 
frames in system is 


Li = gwi = qiw = TL, 


where L (for the aggregate flow) is given by (3.34). Given (3.35), one may be tempted 
to claim that w = L+; to see why this is false (beyond the argument involving Little’s 
formula that was just made), recall the discussion regarding Figure 3.11. 


A system of K servers, no waiting room, and geometrically distributed job interarrival 
and service times is described by a birth-death Markov chain Q as depicted in Figure 2.17 
of Section 2.8 with 

qi = q and p; = ip, 
i.e., a discrete-time M/M/ K/K queue. Therefore, by PASTA in discrete time, the blocking 
probability is given by Erlang’s formula (3.29) with p = q/p. 
Problems 


3.1 In Figure 3.12, cumulative arrivals A and departures D of a packet queue are depicted 
over a period of time in which three packets arrive and depart (both events assumed to occur 
instantaneously for each packet). For example, the third packet arrives at time 4 ms and is 
4 x 50 = 200 bytes long. 


(a) What is the delay experienced by the second packet? 


(b) What is the queue occupancy at time 3.5 ms in terms of bytes and number of packets? 


3.2 Generalize (3.10) to the case where Q(0) > 0. 

3.3 Prove (3.4). 

3.4 Explain why (3.8) does not hold for multiple-server queues. 
3.5 Prove (3.17). 


3.6 Prove Wald’s identity (3.26) for the M/M/1 queue. Also, in the same context, find an 
expression for P(W (T) < x) for x > 0. Hint: Condition on Q(T'—) which, by PASTA, is 
geometrically distributed with parameter p. 


3.7 Find a Markov chain describing the number of jobs in an M/M/K/cx queue. 


3.8 Consider a stationary queue handling jobs with IID exponential service times with 
parameter u. Consider a Poisson process with parameter A < y./2. Suppose that at each 
point of the Poisson process two jobs arrive at the queue. This is a simple "batch" Poisson 
model of the job arrival process. 


(a) Draw the transition rate diagram of the queue occupancy. 


(b) Is it time reversible? 
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Figure 3.12 Cumulative arrivals and departures from a queue. 


(c) Show that the distribution of the workload upon the arrival of a typical job is not the 
steady state distribution. 


3.9 Consider a FIFO queue with a single work-conserving server, infinite waiting room, 
batch Poisson arrivals, and mutually independent job service times that are exponentially 
distributed with mean 1/y. The intensity of Poisson batch arrivals is \ and every batch 
consists of three jobs. 


(a) Draw the transition rate diagram of the queue occupancy (number-of-jobs) process, 
Q. 

(b) State the stability (positive-recurrence) condition for the queue. 

(c) Find the probability that the queue is not empty in steady state, i.e., P(Q > 0). 

(d) Is Q time reversible? Justify your answer. 


Now assume that there are two work-conserving servers instead of just one and at most four 
jobs in the system at any given time. 
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(e) Repeat part (a) for this system. 

(f) State (but do not try to solve) the balance equation of state 3. 

(g) Write an expression for the stationary probability that an arriving job is blocked in 

terms of a. 

3.10 Consider a queue with constant arrival rate of À. The arrivals have "random phase" 
0 uniformly distributed over [0, 1/A], i.e., for ¿ > 1, the ¿th job arrives at time 
i-l 

a 





The service time of the ith job is S;, where the S; are IID, ES; < 1/A (i.e., the queue is 
"stable"), and, to make the queue interesting, P(S; > 1/A) > 0. Show that 


EW(t) = E(W(a)—-U)*, 


where ¢ is a typical time (uniformly distributed between two job arrival times), a is a typical 
job arrival time, and U is a uniform random variable on {0, 1/] independent of a and W. 
This is another example where the implications of PASTA fail to hold. 


3.11 Show that the cumulative departures D from a lossless, initially empty, FIFO queue 
with arrival process A and service rate | satisfy 


D(0,t) = min {A(0, 8) —s}+t forallt > 0. (3.36) 


3.12 Consider a stationary M/M/1 queue. 


(a) Find the stationary distribution of the number of jobs and workload in a (stable) 
M/M/1 queue but not in the server (i.e., in the queue proper). 


(b) Find the stationary distribution of the number of jobs and the workload in the server 
of an M/M/1 queue. 


(c) Apply Little’s formula to both the queue proper and the server. 


3.13 Argue that the job departure process of an M/M/1 queue is Poisson. This is Burke’s 
theorem (also known as the "output theorem" [219]). Hint: Consider the queue reversed 
in time and use the fact that the queue occupancy Q is a time-reversible Markov process 
(Problem 2.13 of Chapter 2). Note that Problem 2.16 is a similar “output theorem" for 
M/GI/oo queues. 


3.14 Consider a stationary M/M/1/K queue with a maximum capacity to hold K < œ 
jobs. 


(a) Find the probability that an arriving job is blocked. Hint: The stationary distribution 
is a truncated geometric. 


(b) Find the mean sojourn time of packets that successfully join the queue. 


3.15 Consider the M/E2/1 queue in which the IID job service times are Erlang (gamma) 
distributed with parameters r = 2 and y and the arrival rate is À. If S is distributed as a job 
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service time, we can write S = @, + @2, where ; and @2 are IID exponential "phases"; 
recall Example 1.7. Let X (t) be the number of phases in the queue at time t. 

(a) Argue that X is a Markov process and find its transition rate diagram. 


(b) Given the stationary distribution of this Markov process, find an expression for the 
distribution of the number of jobs in the queue, Q, in steady state. 


(c) Similarly find an "underlying" Markov process for an M/E;/1 queue for k € Zt and 
relate it to Q. 


(d) Repeat part (c) for an E,/M/1 queue and an E2/E2/1 queue. 


Note that interarrival times and service times of queues that are described by phase-type 


distributions, composed of Erlang distributions with different "u" parameters, also lend 
themselves to Markovian analysis [226]. 


CHAPTER 4 





LOCAL MULTIPLEXING 


In this chapter, we begin to consider multiplexing of multiple flows predominantly in the 
context of a single server (single-bandwidth resource). A preliminary section overviewing 
the architecture of Internet routers is followed by sections on deterministic traffic shaping 
and multiplexing of (layer 3) flows of variable-length packets. We conclude with examples 
of "flow-level" multiplexing, including a description of a generic game-theoretic framework 
involving usage-based pricing. 


4.1 INTERNET ROUTER ARCHITECTURE 


Considering the problem of connecting a large number H ~ 10° end systems/hosts, a 
network of point-to-point connections would require an enormous number of links, on the 
order of H?. The presence of routers (IP layer 3 of the protocol hierarchy) and switches 
(Ethernet layer 2) in the network dramatically reduces the number of required links resulting 
in a topology that is scalable with the number of end systems. 

We now give an overview of the architecture of Internet routers. High-end Internet routers 
are extremely large, fast, and complex systems presently having the ability to forward at 
aggregate packet rates in the terabits-per-second range. In Chapter 3, we studied queues 
handling "jobs," where a job can be interpreted as a packet transmission task. We will see 
how packet queues reside in the packet memories of the ingress and egress linecards of a 
router. 
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4.1.1 Big picture of an IP (layer 3) router 


In Figure 4.1, a3 x 3 router R is shown in two equivalent ways. The depiction at the bottom 
using unidirectional links is more convenient for the following discussion of the generic 
architecture of an Internet router. Large modern routers can have on the order of 128 ports 
(i.e., are 128 x 128) with each port operating at optical carrier (OC) 192 (about 10 Gbps) 
giving a total peak throughput of the router on the order of 128 x 10 Gbps or about a terabit 
per second. The linecard and fabric organization of a router are indicated in Figure 4.2. In 
practice, the ingress and egress linecards of the same interface are on a single board and 
share a local host central processing unit (CPU). The local host CPUs are connected to a 
master CPU (large work station) for the entire router. 


4.1.2 Ingress linecard 


Components of an ingress linecard and switch fabric will now be discussed; see Figure 4.3. 
The IP packets are extracted from the Synchronous Optical Network (SONET) or Ethernet 
frames arriving to an ingress linecard by the deframer device. The headers of the packets 
are then examined by a complex network processor (NP) device. On the fly, the NP will: 


e decide how to forward the packet, 


e update statistics of the aggregate flow of the ingress linecard or of the particular flow 
(session) to which the packet may belong [200], 


e enforce any traffic parameters of the flow (e.g., using a token bucket; see Section 4.2 
below), 


e decrement the time-to-live (TTL) field of the packet, and 
è possibly apply a firewall filter. 


Application of a firewall, TTL decrementer, or token bucket policer may result in the packet 
being immediately discarded. A primary function of an ingress NP is to make the local 
forwarding decision of the packet, which simply amounts to deciding to which output 
port the packet will be transmitted. The forwarding decisions are based on the router’s 
participation in the Internet’s packet-routing protocols; see Chapter 6. A label carrying the 
decisions made by the NP is appended to the forwarded packet to inform the other devices 
in the router that will handle it; this label is stripped from the packet prior to its transmission 
out of the router. 

After the network processor, the packet is stored in a packet memory [137] controlled 
by the ingress traffic management (iFM) device. The enqueue operation of the iTM will 
separately queue packets according to their output port destinations; see the packet flow 
demultiplexer (demux) of Figure 4.4. This virtual output queueing (VOQ) prevents head- 
of-line blocking [120]. 

In ingress packet memory, packets can be further grouped (and separately queued) ac- 
cording to other attributes as identified by the NP and stored in the packet’s label. For 
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Figure 4.1 Depiction of connectivity of router R. 


example, we could partition all the packets flowing to a given output port according to 
their IP port numbers or transport layer protocol: Transmission Control Protocol (TCP) or 
User Datagram Protocol (UDP). For the latter case, a three-queue system per output port 
could be used for TCP control packets (SYN, SYN-ACK, etc.), TCP data packets, and UDP 
packets; see Figure 4.5. There are proposals for frameworks requiring a system of queues 
corresponding to different network-layer classes of service (the "diffserv" framework [24]) 
or bandwidth-provisioned label-switched paths [12, 188]. 
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Figure 4.3 Ingress linecard (input port processor). 


Thus, a two-level hierarchical dequeue (scheduler) would be required: quality-of-service 
(QoS) scheduling among the queues per output port (at least to ensure a minimal amount 
of bandwidth per queue) and scheduling among the output ports; see Figures 4.4 and 4.5. 
Scheduling at the VOQ level of the iTM is typically coupled with switch fabric arbitration 
to evacuate packets to their destined output ports as rapidly as possible. Note that, for each 
identified flow (QoS queue), in-order forwarding of packets is expected of the router. Also 
note that all individual queues are implemented as singly linked lists permitting operations 
only at their head (dequeue) and tail (enqueue) [137]. Operations involving insertion or 
deletion of packets in the interior of a queue or search of the queue’s contents (requiring 
implementation with a doubly linked list) are simply impractical at high data rates. 

Switch fabrics are best designed to operate on fixed-length data segments. The ingress 
switch interface device (iSIF) forms such segments from variable-length IP packets destined 
for the same output port. Indeed, the iSIF may have a small segment queue for each output 
port. A packet may be entirely contained within a segment (depending on the packet’s size) 
or it may be divided among multiple segments. 
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Figure 4.4 Virtual output queues of an ingress packet memory. 


4.1.3 Switch fabric 


An N x N switch fabric simultaneously transmits (switches) N segments from the input 
ports to the output ports. To do this and avoid high memory bandwidth requirements (.V 
times that of the ingress packet memory), no two segments being switched at a given time 
should be destined to the same output port. This requirement is eased for switches that are 
not based on memoryless crossbar fabrics (e.g., shared memory, multistage fabrics [100]) 
at the expense of additional latency. Large, distributed crossbar fabrics may have a bank of 
virtual input queues (VIQs) at the interface to each output port in order to deal with certain 
timing-related memory-bandwidth issues. 

A recent reference for interconnection networks, such as switch fabrics. is [56]. The 
process by which segments are selected for transmission into the fabric is called arbitration. 
Packet-level arbitration [154] works on a shorter time scale than flow/connection-level 
arbitration. Significantly complex computations for flow-level arbitration, such as that based 
on Slepian-Duguid [100] or Birkoff-von Neumann decompositions (two-stage switches) 
(37, 38], can be amortized over a longer time scale. More specifically, flows can be taken 
to join and leave the router on a millisecond time scale compared to the sub-microsecond 
time scale of the transmission time of a packet. Packet-level arbitration can be used to more 
dynamically "fill in the gaps” left by flow-level arbitration. 

The switch fabric is also expected to have a multicasting function, i.e., it can simultane- 
ously switch multiple copies of a multicast packet to its (multiple) destined output ports. 
Such local multicasting functions can be composed to create multicast trees that span the 
entire network [139]. Because of the additional constraints of multicasting and their rela- 
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Figure 4.5 Three tributary queues of a transmission VOQ. 


tively limited numbers compared to unicast packets, multicast packets typically have some 
measure of priority over unicast in the fabric arbitration process. We now discuss a heuris- 
tic for arbitration of provisioned time-division multiplexed (TDM) multicast flows given in 
[230]. 

Suppose that there N linecards, i.e., an N x N switch fabric. Assume that the N 
linecards have data paths of R packet-segments per second. Thus, a time slot of the switch 
fabric has duration R+ seconds. Consider a multicast connection j across the fabric that 
has bandwidth allocation that is a multiple of R, n; R. In the following, connection j will 
generate n; identical vectors v each indicating which (single) ingress source port and which 
egress destination ports constitute the multicast connection under consideration. Let A be 
the set of all such vectors (including identical instances of the same connection) of such 
multicast connections. Define T as the least common multiple of n;. Our objective is to 
schedule all vectors v in A at least once every T time slots. 


Definition 4.1.1. Two vectors in A are said to be compatible if they are disjoint, i.e., if they 
have no common input or output ports; otherwise they are said to contend. 


Packet segments from two multicast connections with compatible vectors can be trans- 
mitted in the same time slot through the switch fabric. A multicast time slot assignment 
problem can now be defined: Assuming that there is no overbooking of bandwidth on any 
input or output link, find a schedule that allocates a time slot to each vector v € A once 
every T time slots. 


Theorem 4.1.1. A feasible multicast time slot assignment problem is NP hard. 
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Proof: Construct a graph whose vertices (nodes) correspond to elements (vectors) in A. 
Links connect all pairs of nodes that are compatible with each other. Consider N different 
colors, where we note that N is an obvious upper bound on the number vectors that can 
be scheduled in the same time slot. Consider a set of vectors that are scheduled in the 
same time slot and suppose that a different color is assigned each such vector. Clearly, the 
multicast time slot assignment problem is equivalent to the vertex .V-colorability problem 
[79]: With N different colors, assign colors to the graph so that no two vertices connected 
by a link are assigned the same color. o 


Basically, CBO is a greedy algorithm. The most contentious connection (i.e., the con- 
nection that contends with the greatest number of other connections) is scheduled in the first 
time slot of the frame. The next most contentious connection that is compatible with the first 
one scheduled is then scheduled for the same time slot and so on until no further connections 
can be scheduled in the first time slot. At this point, the degree of contentiousness of the 
unscheduled flows is recoraputed (decremented) to account for those connections that were 
scheduled in the first time slot and the scheduling process is conducted for the second time 
slot of the frame as it was for the first. Other criteria of the connections or the port indexes 
(e.g., out-degree of the connection or residual demand for the switch port) could be used to 
break ties when scheduling the first connection of a time slot. 


E EXAMPLE 4.1 


Suppose a 5 x 5 fabric needs to schedule the following nine connections: 


Connection Input Port Output Port 





Index Index Index 
A 0 12223 
B 0 2.3.4 
C 1 4 
D 1 4 
E 1 3 
F 2 0.4 
G 3 2 
H 4 0.2.3 
I 4 2 





That is, packets from multicast connection H arrive on input port 4 and are (simul- 
taneously) multicast onto output ports 0, 2, and 3. Note that connections C. D. E.G, 
and J are unicast and that C and D could represent a connection that is allocated twice 
the bandwidth of the rest. Also note that five different connections request output 
port 2. Thus, the fabric’s scheduling frame is at least five time slots long. 

The degrees of contention of the connections are given in the following table. The 
contention values immediately prior to each time slot (row) is recorded in each row. 
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An "xz" indicates that the connection has been previously scheduled. The number in 
bold indicates that the corresponding connection is being scheduled in that time slot. 
The largest (residual contention) number in bold for a time slot corresponds to the 
first connection scheduled under CBO. 








Time Slot\Connection Index | A BC ODEFGHEHI 
0 5 8 4 4 5 4 4 6 4 
1 4 z 3 3 4 3 3 5 8 
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That is, connection B contends with all of the other eight connections. So, B 
is scheduled first in the first time slot. In the second time slot, connection H is 
scheduled first and then C, which is compatible with H, is also scheduled. No other 
remaining connections are compatible with both H and C so we go on to the next 
time slot. Note that in the third time slot D is chosen after A as it has highest 
contention (two), even though F is multicast and D is unicast. In Figure 4.6, the 
fabric configurations of each time slot are depicted. Again, the fabric cycles through 
these five configurations in sequence and each connection therefore receives at most 
one-fifth of a link’s bandwidth R through the fabric. 


4.1.4 Egress linecard 


The egress SIF (eSIF) device of an egress linecard (see Figure 4.7) is for packet reassembly, 
including simple extraction, from the fabric segments. Reassembly is sometimes handled by 
the egress traffic management (eTM). In some routers, there is an egress network processor 
(eNP) to reclassify packets for the purposes of the egress packet memory and output link, 
i.e., in the context of the flows being multiplexed on the output link. The egress packet 
memory, controlled by the eTM device, stores variable-length IP packets. Upon dequeuing 
from the egress packet memory, the framer device forms SONET or Ethernet frames for 
the purposes of transmission onto the output link. 

The egress packet memory may have an organization similar to that depicted in Figure 
4.5 for the ingress linecard with the VOQs replaced by queues feeding separate bandwidth- 
provisioned "channels" partitioning the output link (such channelization is common at links 
crossing boundaries of networks owned by separate entities). So, bandwidth scheduling 
occurs among the channel queues. Scheduling among the class-of-service (CoS) queues 
feeding into a channel could be a hybrid of bandwidth and priority. 
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Figure 4.6 Example schedule of fabric configurations. 


4.2 TOKEN (LEAKY) BUCKETS FOR PACKET TRAFFIC REGULATION 


Suppose that one network provider wishes another to carry a certain flow of packets. The 
network providers strike a service-level agreement (SLA) wherein the transmitting network 


120 LOCAL MULTIPLEXING 








fabric eTM + SONET 
segments eSIF A packet memory framer frames 
1 


A 
l 1 
| i 
| l 
T T 
P 

















labeled IP packets IP packets 


Figure 4.7 Egress linecard (output port processor). 


agrees that his or her packet flow will conform to certain parameters. A preferable choice 
of flow parameters would be those that are: 


e significant from a queueing perspective, simply to ensure conformity by the sending 
network, and 


e simple to police by the receiving network. 


We have seen how useful the mean arrival rate (typically denoted by A) is in terms of 
predicting queueing behavior. The mean arrival rate is, however, difficult to police as it is 
only known after the flow has terminated. Instead of the mean arrival rate, we consider flow 
parameters that are policeable on a packet-by-packet basis. Suppose that when the flow of 
packets arrives to a dedicated FIFO queue with a constant service rate of p bytes per second 
(Bps), the backlog of the queue never exceeds c bytes. One can define o as the burstiness 
of a flow of packets as a function of the rate p used to service it. Such a definition for 
burstiness informs a node so that it can allocate both memory and bandwidth resources in 
order to accommodate such a regulated flow. Moreover, by limiting the burstiness of a flow, 
one also limits the degree to which it can affect other flows with which it shares network 
resources. Indeed, such regulation was standardized by the ATM Forum and adopted by 
the Internet Engineering Task Force (IETF) [94, 95]. 

Suppose that at some location there is a flow of packets A specified by the sequence of 
pairs (T;, li), where T; is the arrival time of the ith packet in seconds (T;+1 > T;) and l; is 
the length of that packet in bytes. More precisely, if c Bps is the peak rate of the medium 
of communication and T; is the arrival time of the leading edge of the ¿th packet, then we 
require that Tj, > T; + l/c, where T; + l;/c is the time at which the ith packet has 
completely arrived. The total number of bytes that arrives over an interval of time (s, t] is 


A(s,t] = do min{l, (t = T)e}l{s < T; < t} 


t 
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Figure 4.8 A token bucket traffic shaper. 


In the following, we will assume c = œ for simplicity. This gives 
Alst) = XO LHs <T; <t}. 
$ 


Assume that this packet flow arrives to a token bucket mechanism as depicted in Figure 4.8. 
A token represents a byte and tokens arrive at a constant rate of p tokens per second to the 
token bucket which has a limited capacity of ø tokens. A (head-of-line) packet i leaves the 
packet FIFO queue when /; tokens are present in the token bucket; when the packet leaves, 
it consumes l; tokens, i.e., they are removed from the bucket. Note that this mechanism 
requires that g be larger than the largest packet length (again, in bytes) of the flow. 

Let A,(s, t] be the total number of bytes departing from the packet queue over the interval 
of time (s, t]. The following result follows directly by consideration of the maximal amount 
of tokens that can be consumed over an interval of time. It holds for all arrival processes A 
to the packet queue. 


Theorem 4.2.1. For all times s < t, 


A,(s,t] < o+p(t—s). 


Any flow A that satisfies the statement of Theorem 4.2.1 is said to satisfy a (ø, p) 
constraint [50]. In the jargon of [94, 95], p could be a sustained information rate (SIR) and 
o a maximum burst size (MBS). Alternatively, p could be a peak information rate (PIR), in 
which case o would usually be taken to be the number of bytes in a maximally sized packet. 

Let Q(t) be backlog at time t of a queue with arrival flow A and a dedicated server with 
constant rate p. 
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Theorem 4.2.2. The flow A is (a, p) constrained if and only if Q(t) < o forall time t. 


Proof: The proof follows from the fact that the maximum queue size is 


max Q(t) = max max { A(s, t] — p(t- 8)}. 





Substituting the inequality in Theorem 4.2.1 gives the result. 











We have shown how the token bucket of Figure 4.8 can delay packets of the arrival flow 
A so that the departure flow A, is (ø, p) constrained. This is known as traffic shaping. The 
receiving network of the exchange of flows described above may wish to: 


e shape the flow using a (g, p) token bucket, or 


e simply identify (mark) any packets that are deemed out of the (ø, p) profile of the 
flow, or 


e drop any out of profile packets. 


The last two actions are known as traffic policing. There are two main devices used for 
traffic policing. The first is a token bucket device like that of Figure 4.8 but without the 
packet queue. A packet is dropped or marked out of profile if and only if there are not 
sufficient tokens (according to its length) in the token bucket upon its arrival. 

Alternatively, by Theorem 4.2.2, one can use a policer as depicted in Figure 4.9. A 
packet is dropped or marked out-of-profile if and only its arrival and inclusion in the virtual 
queue would cause its backlog Q to become larger than a; when this happens, the arriving 
packet is not included in the virtual queue. Note that the virtual queue can be maintained 
by simply keeping track of two state variables: the queue length, Q, upon arrival of the 
previous packet and the arrival time, a, of the previous packet. Thus if a packet of length / 
bytes arrives at time T and is admitted into the virtual queue, then 


Q <— max{Q-—-p(T—a),0}41 
and 
a <— T. 
This (event-driven) operation requires one multiplication operation per packet. Alterna- 


tively, one could maintain the departure time d of the most recently admitted packet instead 
of the queue occupancy Q; see Problem 4.3 at the end of the chapter. 


4.3 MULTIPLEXING FLOWS OF VARIABLE-LENGTH PACKETS 


Suppose that at some location N flows are to be multiplexed (scheduled) into a single 
flow. The flows are indexed by n € {0,1,..., N — 1}. Suppose each flow is assigned its 
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Figure 4.9 A (c, p) constraint policer. 
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Figure 4.10 Multiplexing N packet flows. 


own tributary queue and the output flows of the tributary queues are multiplexed into the 
transmission queue; see Figure 4.10. How the multiplexing occurs depends on the kinds of 
relative priorities the flows possess. 
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4.3.1 Multiplexing with a single FIFO queue 


In this case, the method of multiplexing the queues would be based only on the arrival time 
of the packets to the head of their tributary queue: Packets that arrive earlier are served first 
by the scheduler. Here and in the following, ties are broken arbitrarily. 

In the TCP/UDP context of Figure 4.5, this method has at least one serious problem. 
During periods of congestion, the TCP data flows will back off (congestion avoidance 
phase), leaving the bandwidth resources to the remaining unresponsive (to congestion) 
sessions, which is "unfair" to the responsive TCP sessions [151]. 


4.3.2 Strict priority 


In strict priority multiplexing, flows are ranked according to priority. A flow is served by 
the scheduler only if no packets of any higher priority flows are queued. Even when the 
volume of high priority traffic is limited, there remains the potential problem of long-term 
starvation of bandwidth to lower priority flows. 

The problems with both priority and single FIFO multiplexing can be solved by using 
a scheduler that can allocate a minimal amount of bandwidth to a flow in order to prevent 
long-term starvation of bandwidth to that flow. Bandwidth may also be allocated to flows 
without clear priority designations (as in Figure 4.5) or to flows that are explicitly provi- 
sioned by, e.g., using a provisioned label-switched path (LSP) under Multiprotocol Label 
Switching (MPLS). Also, it may be desirable to share excess capacity (reserved-but-unused 
plus unreserved capacity) according to certain priority rules, e.g., sharing in proportion to 
allocated bandwidth (rate proportionally fair sharing) [84, 166]. 


4.3.3 Deficit round-robin 


Under round-robin multiplexing (scheduling), time is divided into successive rounds, each 
not necessarily of the same time duration. Suppose that in each round there is a rule allowing 
for at most one packet per tributary queue to be transmitted into the transmission queue. 
A problem here is that flows with large-sized packets (e.g., large file transfers using TCP) 
will monopolize the bandwidth and starve out flows of small-sized packets (e.g., those of 
streaming media). Thus, one might want to regulate the total number of bytes that can be 
extracted from any given tributary queue in a round. This leads to the notion of deficit 
round-robin (DRR) scheduling [215]. 

To describe a DRR mechanism, we need the following definitions. Let Lmax be the size, 
in bytes, of the largest packet and Lmin the size of the smallest. Here, the priority of a flow 
has to do with the fraction f» of the total link bandwidth c bytes per second assigned to it, 
where we do not overbook, i.e., 


N 


J odfa Sut 


n=l 


Each flow is assigned its own FIFO queue by the multiplexer. Finally, let 


rain = min fr + 
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i.e., the minimal allotment of bandwidth to a queue. 

Under DRR, at the beginning of each round, each nonempty FIFO queue is allocated a 
certain number of tokens. Packets departing a queue consume their byte length in tokens 
from the queue’s allotment. Queues are serviced in a round until their token allotment 
becomes insufficient to transmit their head-of-line packet. For example, if a queue is 
allocated 8000 tokens at the start of a round and has six packets queued each of length 1500 
bytes, then the first five of those packets are served leaving the trailing sixth packet in the 
queue and 8000 — 5 x 1500 = 500 tokens unused. 

The nth queue is allocated 


tne 
fin 


tokens at the start of a round (again, if it is not empty), thereby ensuring that at least one 
packet from this queue will be transmitted in the round irrespective of the packet’s size. If 
a queue has no packets at the end of a round, its token allotment may be reset to zero. 

Note that the token allotments per round can be precomputed given the allotments fn, 
which change at a much slower "connection-level" time scale than that of the transmission 
time required for a single packet (Limax/c). One could replace fmin in the token allocation 
rule by the minimum bandwidth allocation among nonempty queues at the start of a round, 
but the result would be a significant amount of computation per round possibly precluding 
a high-speed implementation. 

The following theorem demonstrates that DRR scheduling indeed allocates bandwidth 
consistent with the parameters fn. 


Emax 


Theorem 4.3.1. If the nth queue is not empty at the beginning of each of k consecutive 
rounds, then the total number of bytes Dn(k) transmitted from this queue during these 
rounds satisfies 


# Limax T Liin < D,(k) < 


Jn Lmax _ Lin F k Jn Lmax- 


k-1 pe Lies 
( ) fmin fmin 





Proof: The upper bound is obtained assuming that all allocated tokens are consumed in 
addition to a maximal amount of carryover tokens from the round prior to the k consec- 
utive ones under consideration. The lower bound is obtained by assuming no carryover 
tokens from a previous round combined with a maximal number of unused tokens in the 
last round. o 


If two queues n and m are nonempty at the start of each of k consecutive rounds, this 
theorem can be directly used to show 


Drik) _ h, 


lm == = —; 
r= Dil) fim’ 
see Problem 4.4. That is, DRR is rate proportionally fair. 
The general advantage of round-robin schedulers is that they require a minimal amount 
of computation per packet to be implemented. 
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4.3.4 Shaped VirtualClock 


We will now describe a scheduler that employs timestamps to give packets service priority 
over others [238] but restricts consideration only to packets that meet an eligibility criterion 
[102, 209, 214, 236] to limit the jitter of the individual output flows. This trait, which 
is lacking in DRR, is important for link channelization (partitioning a link into smaller 
channels) at network boundaries where SLAs are struck and policed. 

For all i and n, (n, i) denotes the ith packet of the nth flow. Packet (n, 7) is assigned a 
service deadline d,; and a service eligibility time ¢,,,;. A packet is said to be eligible for 
service at time tif € < t. As with DRR, once a packet is "scheduled" by the multiplexer 
(begins service), its service is not interrupted. Upon service completion of a packet, the 
next packet selected for service will be the one with the smallest deadline among all eligible 
packets. Assuming the queues are FIFO, only head-of-queue packets need to be considered 
by the multiplexing (scheduling) algorithm. Each packet (n,7) has two other important 
attributes: its arrival time an; to the multiplexer and its size in bytes, /;. Under what we 
will hereafter call shaped VirtualClock (SVC) scheduling, 


Eni = Max{dyj—1.Ani}, (4.1) 
li 

dni = Eni tz. (4.2) 
fne 


Note that if the nth flow were instead to arrive to a queue with a dedicated server of 
constant rate f,,c bytes per second, then packet (n, i) would: 


e reach the head of the queue (and begin service) at its eligibility time ¢,; and 
e completely depart the server at its service deadline d,, ;. 


In the following, this queue will be called the nth isolated queue (with dedicated server of 
rate fac and arrivals coupled to those of the nth multiplexed queue). 


Lemma 4.3.1. Just prior to the start time of a busy period of the multiplexer, the aggregate 
eligible work to be done of all N isolated queues is zero. 


Proof: Note that the departures from the isolated queues represent the on/y packets eligible 
for service among the queues served by the scheduler. Thus, one can conceive of the sum 
of the byte departures from isolated queues creating “work to be done" for the multiplexer. 
The departure rate at some time t, rn (t) from the mth isolated queue is either fnec bytes per 
second or zero depending on whether the queue is empty. Thus, 


Pat) SG: (4.3) 


That is, the multiplexer is working at least as fast as the servers of the isolated queues at 
any given time. The eligible work to be done by the multiplexer is always less than the 
aggregate of that of the servers of the coupled isolated queues. o 


Now recall Lmax is the maximum size of a packet (in bytes), a quantity that is typi- 
cally about 1500 in the Internet. The following theorem demonstrates that SVC schedules 
bandwidth appropriately in our time-division multiplexing context: 
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Theorem 4.3.2. For all n. and i, the time at which packet (n,i) completely departs from 
the multiplexer of the ith packet of the nth flow is not more than 


dni + Lmax 

c 
Proof: Let 3 be the start time of the multiplexer (mux) busy period in which packet (n. 7) 
(chosen arbitrarily) is served. The previous lemma implies that¢,,; > 3 (otherwise eligible 
work would be available to the mux at time — and, therefore, a mux busy period would 
not start at 3). 

Case 1: First suppose that, during this busy period, no packet having deadline > d,, ; is 
scheduled by the mux prior to packet (n, i). The maximum amount of eligible packets with 
service deadlines < dn. that can be generated by the N queues (flows) over the interval 
13, dyi] is less than or equal to 


y facldn,s a 3] < cldn; = 3] bytes, 


n=l 


where this upper bound is the work performed by the mux over the same time interval. This 
work includes packet (n, i) itself, i.e., the packet departs the mux by its deadline. Note that 
the busy period of the mux may in fact end prior dn.: in this case. 

Case 2: Alternatively, suppose that at least one packet. say (m, j) of flow m Æ n, is 
scheduled prior to packet (n, 7) with service deadline dm. > dn.i. For this to be possible, 
Em.j < En i- Consider the largest eligibility time e* among all such packets (m. j). Note 
that the time t* at which this packet completes service satisfies 


+ of: 2* at Lae < Ens ae Limax ; 
c c 
So, at time t*, all eligible packets with service deadlines < d, ; are considered by the mux. 
By Case | above, the departure time of packet (n, i) will be 


<t* + lni Zeni Lmax i laza = dni + Liias 
fne c fre : 


as desired. oO 








Theorem 4.3.2 is a kind of guaranteed-rate result [126] for the SVC multiplexer. The 
SVC multiplexer also has an appealing property of bounding the jitter of every output flow. 
Consider any flow/queue n and note that the ith packet of this flow will have completely 
departed the multiplexer between times £ni +/,,;/cand d,,; + Lmax/c, where l,,;/cis the 
total transmission time of the packet. We can use this fact and the fact that. by definition, 


dni 2 En ,i+1 


to show: 
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Theorem 4.3.3. The cumulative departures from the nth queue of the multiplexer over an 
interval of time |s, t] is less than or equal to fyzc(t — 8) + 2L max bytes. 


The proof is left as an exercise. There is a significant base of literature on scheduling, 
see, e.g., [51], 80, 126, 166, 176, 184, 209, 214, 236]. In particular, a hybrid of weighted 
round robin and SVC, amortizing the scheduling computation of SVC over a connection- 
level time scale, leads to a joint shaping and (deficit round-robin) scheduling mechanism 
for flows of variable-length packets [111]. 


4.4 SERVICE CURVES 


An alternative formulation of guaranteed service to that of Theorem 4.3.2 is given by the 
service curve concept on which a kind of network "calculus" is based for determining delay 
and jitter bounds for a packet flow as it traverses a series of multiplexed FIFO queues, each 
of which may be shared with other flows. 

Consider a queue occupancy process Q with cumulative arrivals A. Assume that the 
queue receives a service rate of exactly p bytes per second. In [232], it was said that A 
has generalized stochastically bounded burstiness (gSSB) with bound f at p if, for all time 
t > 0, 


P(Q) 20) < f(a), (4.4) 


where f > 0 is a nonincreasing function with f(0) = 1 and, as before, Q(0) = 0 and, for 
t>0, 


Q(t) = max A(s,t] — p(t — s). 


We denote A < (p, f) if A has gSSB with bound f at p. Note that this definition reduces to 
the (ø, p) constraint when f (o) = 0 and recall that, unlike the gSBB, the (ø, p) constraint 
is policeable on a packet-by-packet basis. 

Now consider a scheduler acting on a queue with cumulative arrivals A so that the 
cumulative departures D from that queue satisfy 


D(0,t] > min A[0, s)+ p(t — s) 


0<s 


Re A{0,t — s) + 8(8) (4.5) 


for all time ¢ > 0 and all arrivals A such that the queue is always backlogged over the 
interval [0, t]. In this case, the scheduler is said to offer a service curve 8 [52] to the flow, 
where 3 is a nondecreasing function such that 3(0) = 0. With regard to (4.5), recall the 
expression for cumulative departures of a FIFO queue in Equation (3.36). The following 
theorem shows how the gSBB of a flow changes as it passes through a queue with a service 
curve. 
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Figure 4.11 Output gSSB of a service-curve scheduler. 


Theorem 4.4.1. For a queue with service curve 3, if A<(p, f), then D & (p.g), where 
Sry: Sf (e + min{3(s) + ot}) 


Proof: Consider the backlog of a queue Q whose arrivals are D. That is, consider Figure 
4.11, in which the first queue is the scheduler giving a service curve (2. 
By (3.15), the queue occupancy 


Qt) = ax D/O. 4] — D[0, s] — (t — s)p 


<s<t 


< wax A[0. t] — (ii, A[0. u) + 8(s — u)) —(t-—s)p 


max A[0,t—s+u)—p(t—-s+u)+pu—3{u) 


O<u<s 


= max 
O<s<t < 


< Q(t) + max p(u) — B(u). 
u 
Applying this inequality to the definition of gSSB (4.4) proves the theorem. oO 


The scheduler SCED+ [51] was designed to achieve output service curves. Also, suppose 
a scheduler has guaranteed-rate property parameter u (SVC has u = Lmax/¢ by Theorem 
4.3.2) for a queue with bandwidth allocation p. It is straightforward to show the scheduler 
has service curve 


Ət) = max{pt — pp, 0}. (4.6) 


The proof of this statement is left as an exercise. 


4.5 CONNECTION MULTIPLEXING ON A SINGLE TRUNK 


In this section we consider circuit switching involving a single link with capacity ¢ circuits 
and N users. The nth user has connection setup requests according to a Poisson process 
with intensity A,, and exponentially distributed connection holding times with mean 1/1, 
and each connection requires kn circuits. This is the simplest example of a multiservice 
loss network; see Chapter 5. 
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Theorem 4.5.1. The connection blocking probability of the nth user in steady state is 


TA G(c — kn) 
Giochi" 
where 
Pn = af 
n — Un 3 


and the partition function G is defined as 
pEr 
a = STI a 
zES(c) n=1 
with state space 


N 
S(c) = E E (ZHY | YO knta < e). 


n=1 


Proof: One can first prove by detailed balance that the invariant distribution of the system 
is 





a(x) 


N tn 
pa 
ag US 


see Section 5.1. Note that the numerator in the claimed blocking probability is G(c) — 
G(c — kn) which involves summation over state-space 


n=1 


N 
S(c)\S(e—kn) = [2e 09" le-m < Soham sel, 


i.e., the claimed blocking probability sums over all system "blocking" states x for the nth 
user (states which do not involve kn unoccupied circuits). The theorem then follows by 
PASTA. m 


This blocking probability lends itself to efficient calculation via the Kaufman-Roberts 
iteration; see [45, 189] and Chapter 18 of [186]. 


E EXAMPLE 4.2 


In Figure 4.12, the state spaces S(c) and S(c)\S(c — kn) are depicted for N = 2 
classes, i.e., n € {1,2}, with ky = 1 and kz = 2. 


Multiprotocol label switching [188] enables the establishment of bandwidth-provisioned 
label-switched paths (LSPs) between routers in the Internet. A unidirectional LSP is set 
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Figure 4.12 State spaces for a two-class trunk. 


up from an ingress label-edge router (LER) to an egress LER through a number of label- 
switched routers (LSRs). RSVP-TE [12] or CR-LDP can be used in the setup of bandwidth- 
provisioned LSPs. Once an LSP is set up, specified flows of packets, collectively called 
the forward equivalence class (FEC) of the LSP, are identified by the ingress LER and 
labeled prior to transmission to the next LSR on the LSP. Instead of OSPF or BGP, local 
forwarding (destination output port) decisions by LSRs for such packets are quickly made 
by inputing the labels carried in their headers to MPLS forwarding tables. In the process, 
the labels of the packets are changed ("switched") for the benefit of the next LSR handling 
the packets on their LSP. Thus, the MPLS forwarding tables in the LSRs collectively define 
the routes of the LSPs [188]. The LSR that is the egress LER of the LSP forwards exiting 
packets according to the native routing protocols in the Internet (OSPF and BGP). MPLS 
is discussed further in Section 6.1.2. 
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4.6 A GAME-THEORETIC FRAMEWORK FOR MULTIPLEXING PACKET 
FLOWS 


In this section, we begin to explore how resource pricing may be used to control user demand. 
In our context, the commodity is bandwidth and, by dynamically modifying (increasing) 
prices for bandwidth, one can control congestion (excessive demand). The introduction of 
such mechanisms faces many obstacles, including the deployment of corresponding billing 
technology and unclear commercial advantages over the present system of largely flat-rate 
(unlimited-use) charging for end-user access [162]. However, the future commercial and 
security advantages of resource overprovisioning strategies together with flat-rate pricing 
are not clear either. Moreover, many protocols for resource sharing are very vulnerable to 
misuse by selfish users; pricing mechanisms have been proposed to create the right kind 
of incentives for conformant user behavior [202]. So, the use of price controls is currently 
receiving a great deal of attention both in the literature by researchers and among service 
providers. 

We now study the existence of equilibrium behavior of a population of N users competing 
for resource units (bandwidth) which are offered for sale by a network. We formulate a 
generic noncooperative game that can be applied to several local area networking situations. 
In the following, no explicit negotiation between the users and network are assumed. The 
users are informed of the network’s current pricing, infer their received qualities of service, 
and take action based on their needs and willingness to pay. The network infers total user 
demand (by, say, monitoring queue backlogs) and adjusts its prices accordingly. 

Suppose that each user n has a control (or "access" or "input") parameter Àn and receives 
a certain QoS @,,(A), where 


à = (A1, A2; e AN) 


i.e., the nth user’s QoS depends on the action taken by other users. The QoS may involve 
transmission latency, packet loss rate, throughput, or may simply be defined as an amount 
of bandwidth allocated to the user by the network. We assume that the network charges a 
fixed amount M per unit resource consumed. Each user n seeks to maximize his or her net 
utility or net benefit, in this case 


Un (0n) ~ Mn, 


where Un is the utility function for the nth user. 
Assume the following: 


(a) An € [0, A™?*] for some fixed finite A”°* > 0, 

(b) n(A) is an increasing and differentiable function of An, 
(c) 9n(-) is continuous in TI 0, Amex], 

(d) n(A) = 0 for all A such that Àn = 0, 

(e) Un is increasing and Un (0) = 0, and 

(Ð Us, is decreasing, i.e., Un is concave. 


Examples of simple functions 8 representing network dynamics are the subject of the 
following example and Problem 4.11 at the end of this chapter. 
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E EXAMPLE 4.3 


Suppose that the user under consideration is willing to pay m dollars for each suc- 
cessfully transmitted byte (of a particular application) interpreted in the following 
way: When M = m, the user will desire only a throughput of ©. When M > m, 
the user’s desired throughput, y*(M) = 0. Over the interval 0 < M < m, y*(M)is 
decreasing and y* (0) = 7, where z is the maximum possible throughput requirement 
of the associated application. The following is an example formula for a user’s desired 
throughput consistent with the previous discussion and parameterized by m, O, 7: 


gdn if M>m, 
Y 7 min{mO/M, x} if 0O< M<m 


(clearly, demand y* can decrease with increasing price M in other ways). In practice, 
the user will try to choose his or her throughput @ so as to maximize his or her net 
benefit U (0) — M9 (utility minus cost). The maximizing value of 6 is (U’)~' (MZ). 
Equating with (4.7) gives 


Sir = FIA Ai PN one 0 if Af >m, 
GN) es) E l min{mO/M, n} if OSM<m. 
Therefore, (4.7) corresponds to the utility function 
mo if 0<0<0. 
UUP = mO(log(@/O)+1) if O<6@<7, (4.8) 


mO(log(7/O)+1) if 7<@. 


This utility U is concave and nondecreasing. Also, U is strictly increasing on 
[0, œ) only when 7 = œ. Moreover, U is strictly concave on [0, oc) only when 
© = Qand 7 = ox, which is the case where the user’s QoS requirements are "elastic" 
[124, 201], i.e., the utility gradually increases but sublinearly as a "law of diminishing 
returns.” See Figure 4.13. 


Assumption (f) implies arg maxg{U;,(9) — MO} is unique; therefore, let 
yn = (Up) (M) 


and assume that yn > 0. The nth user sequentially adjusts his or her control parameter An 


(4.7) 


in an attempt to make the received QoS, 6,,(A), equal to yn. The following theorem is an 


immediate consequence of Brouwer’s fixed-point theorem [27, 205]. 


Theorem 4.6.1. Ifa jointly continuous function G satisfies the property 


=r ify=46 
Gly,6,rA)< >A ify>s (4.9) 
<à ify <4, 


then the iteration in j 


NT = min {G(Yn-On(M?),M), ABO}, DE {1,2 NL None 
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Figure 4.13 Example of user utility function. 


has a fixed point. 


The function G represents the end user’s rate control strategy of the game (4.10). Ex- 
amples include multiplicative increase and multiplicative decrease, i.e., 


G(y,0,d) = Ay (4.11) 
and additive increase and multiplicative decrease (as with TCP). Note that we are implicitly 
assuming that every user does not divulge his or her utility function and, in particular, does 
not divulge their demand yn for a given price M. This fixed-point iteration requires only 
that at each step 7 the users ascertain (estimate/assess) their own received QoS, 0, which 
results from their choice of control parameter, A. Thus, we assume that the "network" 
dynamics for fixed control parameters are fast compared to the slow time scale of the 
iteration (4.10) allowing the users to observe the steady-state results of their actions. Were 
the network to adjust its price(s) M (up or down according to the measured level of user- 
demand (congestion) ), it would, in turn, do so on a slower time-scale than that of the user 
iteration (4.10). 


Definition 4.6.1. A fixed point X* is said to be a Nash equilibrium point (NEP) if, for all 
users n, the net utility 


Un (On (A)) = Môn (à) 
is maximized at An = X, assuming Aj = A} for all j # n, ie., for all n, 


à = ag TE Un (O(A; à” n) — MOA; A” n), 
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where (A; A*_,,) is the N-vector whose kth component is \; for all k + n and whose nth 
component is À. 


Note that at a NEP there is no incentive for a user to unilaterally change his or her action 
while the others’ actions remain fixed. The following simple theorem concerns the fixed 
points A* of the generic iteration (4.10). 


Theorem 4.6.2. For the game (4.10) of Theorem 4.6.1: 
(a) If <A, On(A") = Yn 
(b) On(A") < Yn 
(c) Each such fixed point is a NEP. 


Proof: (a) and (b): From (4.10), A* equals either A™** or G(yn,On(A*), Až), which is 
smaller than Ape”. If 


An = Gyn, On(A"), An) < An, 
then (a) follows from the definition of G. If Až = A™**, then 
G(yn, On(A*), An) 2 Agee 


Hence, n (A*) < yn holds. 

(c): Given the continuity assumption of each @,, and the definition of yn, it is clear that 
all fixed points in the interior of Ie 30: Amax] are NEPs. By (a) and (b), to show that 
a fixed point A* on the boundary of TL, (0, Amax] (i.e., some of whose components are 
Apax for some n) is a NEP, we will show that if 


Ap So AES, (4.12) 
then 
Ne tE À 
h ag ate y FA), 
where 


FA) 


This statement clearly follows if f’ > 0 (f is nondecreasing) over the whole interval 
[0, Apax], By direct differentiation, 


Un (On (A; A” n)) — MOn(A;A* n). 


abn (A; A” n) 


FA) = (UL, (On (à; à*n)) — M) ar 


First note that 06,,(A; A*_,,)/OAn > 0 by assumption. By statement (b), 
Onl“) <S Yn = (UL) (M ); 
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therefore, because U’ is nonincreasing by assumption, 
Tl 


U (On A" )) = UnA An) = M. 


LAT = 


Again, because U/, is nonincreasing and because @,, is nondecreasing in \,, (also by as- 
sumption), the previous inequality implies 
UL Onl A n) > M 


for all A < AP as desired. o 


E EXAMPLE 4.4 


Recall the ALOHA wireless random multiple access system in Problem 1.22 of Chap- 
ter 1. Now assume that the users adjust their retransmission parameters An (i.e., they 
are not all the same fixed value p) and that the QoS of a user is their average throughput 
Le., 


On(4) = An [0-A 
issn 
Consider the game (4.10) with update function (4.11) for this ALOHA context with 


Nee ài 1 
for all n, and consider a NEP A*. 
First, if A* solves 
Yn > 
àn = = forall <n<N (4.13) 
Hien G ms Ai) 


and 0 < Ay < 1 (ie, AF € (0,1)) for some n, then 0 < Af < 1 foralli Æ nas 
well. Second, if, for some n, the polynomial in Àn 


An(1 — An)! = 9m [] (1 + (# - 1) an) (4.14) 


isn Yn 


has a real root à% € (0, 1), then (4.13) has a real solution A* in (0,1). 


Lyapunov stability methods were applied to study the limiting behavior of the generic 
fixed-point iteration for ALOHA [109] and single-trunk Erlang loss systems [110]. A 
Markov chain describing an ALOHA system on a graph (spatially interfering access points) 
was studied in [26]. Several authors have also studied dynamic pricing mechanisms in a 
networking context; see Chapter 6. Increasing but nonconcave utility functions (i.e., S- 
shaped functions approximating a discontinuous step) are indicative of users with inelastic 
QoS requirements; see [140]. Note that in our problem formulation above, U(@(-)) may be 
a nonconcave function. 
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4.7 DISCUSSION: LOCAL MEDIUM ACCESS CONTROL OF A SINGLE 
WIRELESS CHANNEL 


Carrier sense multiple access with collision avoidance (CSMA/CA) [22, 33] systems reduce 
collisions (interfering packet transmissions) via channel monitoring and interframe spacing 
rules, and also use windowing-based backoff random access in response to collisions [139]. 
Hybrids of random-access and round-robin scheduling have also been proposed for time- 
division multiple access (TDMA) of a wireless channel. A vast literature exists for non- 
TT oA "spread-spectrum” methods, including those based on code-division multiple access 
(CDMA) and pseudorandom frequency hopping (as in Bluetooth); see Problem 4.12 below. 


Problems 


4.1 Start with the expression (3.3) from Chapter 3 and assume (ø, p) constrained arrivals 
to obtain a bound on EQ and, thereby, on P(Q > M). 


4.2 Suppose a flow is both (ø, p) and (L, m) constrained with p < 7 bytes per second and 
ao > L bytes. Such a constraint can be achieved by a dual leaky bucket (two leaky buckets 
in series) [4, 5]. 


(a) Find a tight upper bound on the cumulative transmission (in bytes) of the flow over 
an interval of time of length t [95]. 


(b) Find the smallest b such that the flow is (b,c) constrained for a given c such that 
p<c< 7. Hint: Plot the line ct with the answer to part (a). 


The queueing behavior of such dually constrained flows was studied further in [128]. 
4.3 Consider the policer of Figure 4.9. 


(a) How would it function if it were to maintain the departure time of the most recently 
admitted packet d instead of the queue length Q. Is the state variable a required? 


(b) Look up the packet markers defined in [94, 95] and determine whether they are 
implemented using the mechanism in Figure 4.8 (without the packet queue) or Figure 
4.9 and what state variables are maintained. 


4.4 Under DRR, suppose two queues n and m are nonempty at the start of each of k con- 
secutive rounds. Find expressions for upper and lower bounds on the ratio Dn (k)/Dm(k) 
and show that this ratio converges to fn/fm as k — œ. 


4.5 Prove Theorem 4.3.3. 


4.6 Show that, under SVC, the service curve D of a queue with bandwidth share f is 


Dy(t) = max{fet — Lmax. O}. 


4.7 Prove the statement of (4.6). Hint: Suppose the service deadlines of an arrival process 
{A, a} are d for a bandwidth allocation p, i.e., definition (4.2). Let d be the departure times 
from the scheduled queue under consideration so that the cumulative departures from the 
queue are 


Do.) = Xuda <t} > SO Hadrut} 


nN n 
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where the last inequality is the assumed guaranteed rate property. 


4.8 Prove a guaranteed rate property like that of Theorem 4.3.2 but for the basic Virtu- 
alClock scheduler that does not use eligibility times, i.e., the eligibility time of a packet is 
simply its arrival time. Note that VirtualClock does not possess the same per-flow output 
jitter bounds of SVC. 


4.9 A concave cost function M (0) could be used instead of a proportional cost A/6 to 
encourage additional user consumption with volume discounts. If U}(@) — Af’(@) is a 
decreasing function of 6, then we have a unique maximizing value of U,, — M: 


Yn = arg max{Un (0) — M(6)}. 


If \(6) is a concave differentiable function of @ with M (0) = 0, U2 (0) > M'(0), and 
U’, — M' decreasing, determine whether Theorem 4.6.2 continues to hold with yn replaced 
by % for all n. 


4.10 Prove the statements of Equations (4.13) and (4.14). 


4.11 Find an expression for the QoS 6,,(A) used in the generic game (4.10), where the 
users share a single trunk of Œ circuits, assuming each user controls the intensity A» of 
their connection setup request process submitted to the network, the average holding time 
of a connection of the nth user is 1/jin, and the QoS of a user is the steady-state number of 
active connections they hold. 


4.12 Suppose that there are K channels and N < K nodes. The nodes are synchronized 
and time is slotted. In each time slot, each node selects a channel at random and attempts 
to transmit a single packet, i.e., a frequency-hopping spread-spectrum system. Find an 
expression for the probability that one or more transmission collisions occur in a time slot. 


4.13 This problem explores a simple power control iteration for the uplink of a wireless 
cellular network. Consider an end system j whose base station is denoted bj. Let hy, , be 
the path gain from node k and pj, be the transmission power of node k after 7 iterations of 
the following algorithm. Let R, be the desired transmitted data rate of node k, W be the 
channel bandwidth, and g? be the background Gaussian noise power. The throughput of 
node j’s uplink (from node 7 to b;) is 


nt 
7 we 
i 3 

o? + ey hb; k Ph 

where ~; is a parameter that weakly depends on j [197]; note the signal-to-interference- 
and-noise ratio in this expression. Assume end-system mobility is negligible (and hence 
the hy, % terms are constant) during the iteration 





2 A 
p = SR + egg hej kPe 
i ARES A 


(a) Show that the iteration can be written, in vector form, as 
ale = Ap’ + 8, 


i.e., find the square matrix A and vector s. 
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(b) Find a condition on A so that a unique fixed point p* exists. 


(c) Find a condition on A so that p converges to p*. 
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CHAPTER 5 





QUEUEING NETWORKS WITH STATIC 
ROUTING 





In this chapter, we introduce two classical stochastic queueing models: "loss" networks 
modeling circuit-switched networks and Jackson networks that can be used to model packet- 
switched networks and packet-level processors. Static routing is considered in the former 
and purely randomized routing with static routing probabilities is considered in the latter. 
Dynamic routing is the subject of Chapter 6. 


5.1 LOSS NETWORKS 


In this section, we consider circuit-switched loss networks for which the basic references are 
[123, 189]. The traditional telephone system is circuit switched. Also, there is significant 
MPLS deployment in the Internet today allowing LSPs (similar to ATM "virtual" circuits) 
to connect different access points of an Internet service provider (ISP). Circuit-switched 
frame relay continues to handle a substantial amount of virtual private network (VPN) and 
leased-line service. 
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Figure 5.1 An example loss network. 


E EXAMPLE 5.1 


Consider Figure 5.1 depicting a network with 13 enumerated links. Note that the 
cycle-free routes connecting nodes (end systems) m and n are 


rı = {1,3,6,9}, 
ro = {1,4,13,9}, 
r3 = {1,4,5,6,9}, 
ra = {1,3,5,13,9}, 


where we have described each route by its link membership as above. We will return 
to this example in the following. 


Consider a network connecting together N end systems/users. Bandwidth in the network 
is divided into fixed-size amounts called circuits, e.g., a circuit could be a 64 kbps channel 
(voice line) or a TI line of 1.544 Mbps. Let L be the number of network links and let c; 
circuits be the fixed capacity of network link l. Let R be the number of distinct bidirectional 
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routes in the network, where a route r is defined by a group of links / € r. Let R be the 
set of distinct routes so that R = |R|. Finally, define the L x R matrix A with Boolean 
entries a; in the lth row and rth column, where 


1 ifler, 
0 ifl¢r. 


That is, each column of A corresponds to a route and each row of A corresponds to a link. 


alr 


E EXAMPLE 5.2 


The four routes rı to r4 for the network of 13 links given in Example 5.1 are described 
by the 13 x 4 matrix 





By vb a 
0 0 0 0 
1 0 0 1 
0 1 1 0 
0 0 1 1 
1 0 1 0 
A = 0000 
0000 
1 1 1 1 
0000 
0 0 0 0 
0 0 0 0 
0 1 0 1 


5.1.1 Fixed-route arrival rates 


In this section, we assume that each route has an independent associated Poisson connection 
arrival (circuit setup) process with with intensity à». This situation arises if, for each pair of 
end nodes 7, there is an independent Poisson connection arrival process with rate A, that is 
randomly thinned among the routes Ry connecting 7 so that there are independent Poisson 
arrivals to each route r € R, with rate pr.r > 0, where these fixed "routing" parameters 


satisfy 
5 Prr = 1. 
TER» 


Define X,(t) as the number of occupied circuits on route r at time t and define the 
corresponding random vector X(t). Let 


(5.1) 


Er 


be the R-vector with zeros in every entry except for the rth row whose entry is 1. If an 
existing connection on route r terminates at time t, 


X(t) = X(t-) -en 


144 QUEUEING NETWORKS WITH STATIC ROUTING 


Similarly, if a connection on route r is admitted into the network at time t, 
X(t) = X(t-)+e,. 


Clearly, a connection cannot be admitted (i.e., is blocked) along route r* at time t if any 
associated link capacity constraint is violated, i.e., if, for some l € r*, 


Y X,(t-) = a. 
r|ler 


An R-vector x is said to be feasible if it satisfies all link capacity constraints, i.e., for all 
links /, 


(Az), = pD xy € {0,1,2, ..., q}. 


r | ler 


Thus, the state space of the stochastic process X is 


I 


S(c) {z € (Z7)? | Az < c}. 


E EXAMPLE 5.3 


For the network of Figure 5.1, note that link 1 is common to all routes of Example 
5.1. Thus, 


R 
I 
PRR RP 


is feasible if the capacities cœ = 4 for all links / but is not feasible if cı < 4 because 
(Az), = 4, 


where A is given in Example 5.2. 


In this section, we also assume that for each route r the connection lifetimes are in- 
dependent and exponentially distributed with mean 1/j,. Under this assumption, it is 
easily seen that the stochastic process X is a Markov chain wherein the state transition 
x—+2x+e, € S(c) occurs with rate À, and the the state transition z — x — e, € S(c) 
occurs with rate £y Hr. 





Theorem 5.1.1. The loss network X is time reversible with stationary distribution on S(c) 
given by the product form 





~_ Je 
o(z) = ae Us (5.2) 
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(x, + 1) er 


Figure 5.2 Part of the TRD of a loss network. 


where 


cis the L-vector of link capacities, and 
pr" 
G = — : 
(c) pr Bes (5.3) 
LES(c) TER 


is the normalizing term (partition function) chosen so that zesio) o(z) =1. 


Proof: Assuming z, x +e, € S(c) for some r € R, a generic detailed balance equation is 
Aro(z) = (er + l1)uro(z + e,): 


see Figure 5.2. The theorem statement therefore follows if o given by (5.2) satisfies this 
equation. So, substituting the expression (5.2) for ø and canceling from both sides the 
normalizing term G(c) and all terms pertaining to routes other than r give 


pr purr 
Ay = + 1u. 
zy} (@r Mr (zr +1)! 


This equation is clearly seen to be true after canceling x, + 1 on the right-hand side, then 
canceling p** /x,! from both sides, and finally recalling pr = Àr /Hr. Oo 


Note how G depends on c through the state space S(c) and compare with the definition 
of the partition function in Theorem 4.5.1 for a single link with multiclass connections. 
This result can be generalized to multiclass networks [122, 189] and holds for different job 
service time distributions [224] (i.e., the theorem statement is "insensitive" to service time 
distribution). 


5.1.2 Exact expression for connection blocking 


A connection is admitted on route r at time t only if a circuit is available, i.e.. only if 
(AX(t—))) < & — 1 for alll € r, where (Ax); represents the [th component of the 
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L-vector Ax. Consider the L-vector Ae 


=r? 


i.e., the rth column of A whose /th entry is 


1 ifler, 
(Ae) = k iflgr. 


Thus, the L-vector c — Ae, has lth entry 


q—-1 ifler, 
a= (Ae) = k iflgr. 


Theorem 5.1.2. The steady-state probability that a connection is blocked on route r is 


G(c — Ae, ) 
G(c) 


B: s We (5.4) 


Proof: First note that B, is 1 minus the probability that the connection is admitted (on 
every link / € r). Therefore, by PASTA, 


B, = 1- D a(x) 


Agsc—Ae,. 
1 pz" 
-1-0 oS 
t ? 
Glo) esia Ae) 7 
from which (5.4) directly follows by definition of the normalizing term G. o 


5.1.3 Fixed-point iteration for approximate connection blocking 


The computational complexity of the partition function G grows rapidly as the network 
dimensions (L, R, N, etc.) grow [189]. In this section, we formulate an iterative method 
for determining approximate blocking probabilities under the assumption that the individual 
links block connections independently. Consider a single link /* and let bj» be its unknown 
blocking probability. For the moment, assume that the link blocking probabilities b; of all 
other links | 4 /* are known. Consider a route r containing link /*. By the independent 
blocking assumption, the incident load (traffic intensity) of /* from this route, after blocking 
by all of the route’s other links has been taken into account, is 


pr [J (by). 


ler | lAl* 


Thus, the total load of link /* is reduced! by blocking to 


do JT] G@-&) = ôr), 


r| ler ler | I4l* 


lOr “thinned"; see Problem 2.12 of Chapter 2. 
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where b_, is the (L — 1)-vector of link blocking probabilities not including that of link J. 
By the independent blocking assumption, the blocking probability of link /* must therefore 
be 


be = El pi- (bp). cr), (5.5) 
where € is Erlang’s blocking formula (3.29): 


p/e 
Erö pi / jt 


System (5.5) is known as a reduced-load approximation. 

Clearly, the link blocking probabilities b must simultaneously satisfy Equation (5.5) for 
all links /*, i.e., (5.5) is a system of Z equations in L unknowns. Approaches to numerically 
finding such an L-vector b include Newton’s method and the following fixed-point iteration 
(method of successive approximations). Beginning from an arbitrary initial b, after j 
iterations set 


E(p.c) = Elp) = 


b? = E(ĝ(bÎ7"), ci) for all links Z. (5.6) 


Brouwer’s fixed-point theorem gives that a solution b € [0, 1] to (5.6) exists [27]. Unique- 
ness of the solution follows from the fact that a solution to (5.5) is the minimum of a convex 
function [123, 189]. 

Given the link blocking probabilities b, under the independence assumption, the route 
blocking probabilities are 


B, = 1-][Q@-») 
ler 
= Ta+o(Ta), 
ler ler 


i.e., if $ jep bi < 1, then 


B, x N by. 


ler 


That is, the blocking probability is approximately additive. 


5.2 STABLE OPEN NETWORKS OF QUEUES 


In the previous section, an idealized circuit-switched network was considered in which a 
"job" (connection or call) occupied a circuit on every link of a route and there was no 
waiting room for jobs. The arrival rates to the routes r are fixed A,.. In this section, we 
will consider an idealized packet-switched network where the routing decisions, made at 
each node forwarding the packets (jobs), are random. Also, the service times at the nodes 
that a given packet visits are random and independent with distribution depending on the 
forwarding node. If the network is modeling packets flowing through Internet routers, 
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then this too is somewhat of an idealization considering that all of a given packet’s service 
times would be proportional to its length. A histogram of packet length distributions in 
the Internet is given in [2]. The mean packet service times could be determined from such 
a histogram and the (constant) service rate of the associated node/router. The material in 
this section has applications beyond a Markovian model of a packet-switched network; for 
example, the flow balance equations of Section 5.2.1 below can also be used to model the 
interworkings of the various “engines” within a network processor. 

Consider a group of N > 2 lossless, single-server, work-conserving queueing stations. 
Packets at the nth station have a mean required service time of 1/1, foralln € {1,2,..., N}. 
The packet arrival process to the nth station is a superposition of N + 1 component arrival 
processes. Packets departing the mth station are routed to and immediately arrive at the 
nth station with probability m,n. Also, with probability r,,,9, a packet departing station 
m leaves the queueing network forever; here we use station index 0 to denote the world 
outside the network. Clearly, for all m, 


N 
X Tmn = 1. 
n=0 


Arrivals from the outside world arrive to the nth station at rate An. It is these interactions 
with the outside world that make the network open. 


5.2.1 Flow balance equations 


Let Àn be the total arrival rate to the nth station. These quantities can be found by solving 
the so-called flow balance equations which are based on the notion of conservation of flow 
and require that all queues are stable, i.e., Un > An for all n. By Theorem 3.4.5, the flow 
balance equations are, for all n € {1,2,..., N}, 


N 
An = Ant So Amm: (5.7) 


m=1 


Note that the flow balance equations can be written in matrix form: 


ATR) = AT 


? 


where the N x N matrix R has entry rm,n in the mth row and nth column. Thus, the flow 
balance equations can be uniquely solved if det(I — R) Æ 0, i.e., I — R is invertible so that 


AT = ATU-R)7. 


Again, we are assuming that A < u. This also requires that rm,9 > O for some station m 
(i.e., packets can exit the network). Otherwise, R would be a stochastic matrix (all entries 
nonnegative and all rows sum to 1) so that 1 is an eigenvalue of R and, therefore, O is an 
eigenvalue of I — R, i.e., I — R is not invertible [99]. 

We can also write a flow balance equation between the outside world and the queueing 
network as a whole by summing (5.7) over n € {1,..., N}: 


N 
Sa 
n=1 


N 


do dari (5.8) 


n=l 
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Figure 5.3 An example packet network. 


i.e., the total flow into the network equals the total flow out of the network. 
The flow balance equations hold in great generality. In the following, we will apply 
them to derive the stationary distribution of a special network with Markovian dynamics. 


E EXAMPLE 5.4 


For the example network of Figure 5.3 with three lossless FIFO queues, queues | and 
2 respectively have exogenous arrival rate A, and Az packets per second. The mean 
service time at queue k is 1/4. The nonzero packet routing probabilities are 


= =, 
M120 > T1383 3 
1 
T21 = 123 = 9; 
= = zai 
T31 = 132 = 730 = 3) 


where the subscript 0 represents the outside world. 
Assuming that the queues are all stable, the flow balance equations are 


à = Ar + $A + $da, 
à = Ag+ 4A1 + A3, 
Ae = HA thz 
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Thus, in matrix form, 


1-3} -1 A 
Pee aa d e 
if É EE a 
-4 4 i o 
which implies 

ied = TTF M 

= 1 
as j- 1a] [a 
-3 73 l 0 


ie., A = ((I~R)")7!A. Given the total flow rates À, the service rates ji, can 
be chosen so that uk > Ax for all queues k to achieve stability (the flow balance 
equations hold). Finally, note that the mean departure rate to the "outside world,” A3, 
will work out from the flow balance equations to be 


A3r30 = Ay, + Ag. (5.9) 


5.2.2 Open Jackson networks 


Suppose that the exogenous arrivals are Poisson and that the service time distributions are 
exponential. Further suppose that all routing decisions, service times and exogenous (ex- 
ponential) interarrival times are mutually independent. The resulting network is Markovian 
and is called an open Jackson network. Note that if X, (t) is the number of packets in station 
n at time t, then the vector of such processes X (t) taking values in (Z+)” is Markovian. 

We will now specify a Jackson network’s transition rate matrix and stationary distribution 
on (Z*)%, Consider a vector x = (a1,...,2) € (Zt) and define the following operator 
6 mapping (Z*+)* — (Z+)%. If am > Oand1 < n,m < N, then ôm n represents a 
packet departing from station m and arriving at station n: 


Li ifi £ m,n, 
Omit = Zm -1 ifi=m, (5.10) 
Intl ifi=n, 


i.e., Ôm n£ = £ — Eem + £n- lf £m > Oand1 < m < N, then ôm 9 represents a packet 
departing the network to the outside world from station m: 
ifi £m, 


ifi= n. 


(Om,0 x); = f n 1 


Lm — 


If 1 <n < N, then 69,, represents a packet arriving at the network at station n from the 
outside world: 


_ Ti ifi xn, 
(Son B)i = eae ifi=n. 


The transition rate matrix of the Jackson network is given by the following equations: 





E ae LmTmn fl <m<N,0<n<N, ry, > 0, 
q £, m,n x i= An ifm = 0, 1 < n < N. 
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Theorem 5.2.1. The stationary distribution of an open Jackson network is 


a(x) = 1 TT (5.11) 


n=1 


where, for all n, the traffic intensity 


ll 
| 
A 


Pn 


for stability and the normalizing term (partition function) is 


G = = ee 


E(Z+)N n=1 


Proof: Recall the balance equations (2.29) for the Jackson network "at" x, i.e., correspond- 
ing to x’s column in the network’s transition rate matrix. To do this, we consider states 
from which the network makes transitions into x. These include: 


e bn m x for stations n such that zn > 0, where 


aln, m x, x) = Umm on: 


e 6o.n x for all stations n, where 


Is 
IR 
— 
I 
> 
3 


q(on,o 


e all other transitions that do not occur in one step, i.e., q = 0. 


Note that m, nôn,m £ = x. Therefore, the balance equations at x are 


N 
` e L. x)a(dn.o x) + x alinm T, Z)O(On an 2) 


{n | £n oe m=1 


Se i Ja (o.m £) 


m=] 


= 5 i (z. ina ad qlz, dm.n £) 


{m | tm>0} 


Sue do.n £) | a(x). 


n=1 
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Substituting the transition rates we get 


2 


{n | zn >0} 
N 


+ 5 Lmtm,00 (O0.m z) 


ml 


N N 
= 5 pnt ++ Se mn + 5 An a(x) 


{m | tm>0} n=1 n=1 





N 
Ano(dn,0 x) + 5 Lm?m,n?(bn.m J 


mæl 


Il 


N 
5 lim + 5 An | o(z), 


{m | zm>0} 


where the last equation is (5.7). The theorem is proved if we can show that the distribution 
a given in (5.11) satisfies this balance equation at every state x. Substituting the expression 
(5.11) for o and factoring out o (x) on the left-hand side, we get 


N N 2 
D [hir Enne] Santon = tnt Se 
n=) 


{n | an>0} 7 mel i m=l {m | tm>0} 





Substituting Àm = UmPm, We get 


1 N N N 
Dn (a. Te 5 Mnn J| + 5. ÀmTm,0 = 5 Hm + No An. 
m m=1 n=1 


m=1 {m | 1m >0} 


{n | 2n>0} | 


Finally, substitution of the flow balance equations (5.7) and (5.8) implies this equation does 
indeed hold. oO 


The expression in (5.11) is called a product form distribution. Note that the product form 
implies the queue occupancies are statistically independent in steady state. 


E EXAMPLE 5.5 


Suppose the network of Example 5.4 is Jacksonian. The steady-state distribution of 
the number of jobs in the third queue is 


P(Q3=k) = p3(1—ps), 


where p3 = 43/3 < 1. Thus, EQ3 = p3, which is consistent with Little’s formula. 


Additional details of open Jackson networks and closed (fixed-population) Jackson net- 
works are given in [226]. 
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Figure 5.4 A three-node queueing network. 


3 


Problems 

5.1 Verify the details in the proof of Theorem 5.2.1. 

5.2 Give an intuitive explanation for why (5.9) ought to hold. 
5.3 Is a Jackson network time reversible in general? 


5.4 For the network of three lossless FIFO queues depicted in Figure 5.4, queue 1 has 
exogenous arrival rate A, packets per second. Each packet departing from queue 1 has 
one of three possible fates: It is forwarded to queue 2 with probability 2 and to queue 3 
with probability § and leaves the network (back to the outside world) with probability 3. 
All packets departing queues 2 and 3 reenter queue |. For each queue k (k = 1, 2,3), the 
mean service rate is 4p packets per second and the total mean arrival rate is Ày packets per 
second. 


(a) If all queues are stable, i.e., A, < Hk for all k = 1, 2,3, find A, only in terms of 
A,. So, given Aj, find the smallest service rates jj, such that the network is stable 
whenever uk > u% for all queues k. 


(b) If 1 = 7, u2 = 3, and u3 = 4, find the largest Aj such that the queueing network is 
stable for all Ay < Af. 


Now suppose the network is a Jacksonian and in steady state: 


(c) Find the probability that a packet visits only queue 1 (once) and experiences no 
queueing delay. 


(d) Find the mean total sojourn time of packets in the network, i.e., from the time a 
packet enters into the first queue from the outside world to the time it finally leaves 
the network. 
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Figure 5.5 An open network of three lossless, FIFO, work-conserving queues. 


5.5 For the network of Figure 5.5, half the jobs are routed back into the previous queue, 
so that half the jobs departing the rightmost queue (indexed 3) leave the system. The 
exogenous arrival rate (to the leftmost queue indexed 1) is A jobs per second. 


(a) State and solve the flow balance equations (assuming the queues are all stable). Label 
Figure 5.5 with the total arrival rate \,, to each queue n. 


(b) Find the smallest number j:* for each queue n such that the network (i.e., every 
queue) is stable if un > u% for all n. 


Now assume the network is Jacksonian and in steady state: 


(c) Find the mean sojourn time (queueing delay plus service time) of packets through 
queue 2 if u2 = 8A. 


(d) Additionally if jz; = 5A and u3 = 6A, find the mean sojourn time of a packet through 
the entire network, i.e., from the time it arrives to queue 1 from the outside world to 
the time it completely departs queue 3. 


5.6 Consider the context of Section 5.1.1. 


(a) Define Y;(t) as the number of occupied circuits in link / at time t and define the 
corresponding random vector Y(t). Is Y a Markov chain? Hint: Consider state 
transitions when a connection terminates. 


(b) Show that link l’s traffic intensity is 


pia So 
Hr 
r|ler 


5.7 Consider a queueing system with an arrival process that is a stationary marked Poisson 
process. If the state of the stationary and ergodic (and causal) queueing system at time t, 
X+, belongs to a set of "blocking states” B, a job arriving at the queueing system at time t 
is blocked. Let B, be the number of blocked jobs over [0, t] and let N; be the number of 
arriving jobs over [0, t]. Show that the stationary probability that an arriving connection is 


blocked is equal to 
B; 
E| —-|N, : 
(= | í Z 0) i 
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ie., when N; > 0, B/N; is an unbiased estimate of the blocking probability. Hint: 
Condition on N; and then use the conditional uniformity property to show 


(FIM >0) = P(X eB), 


the stationary probability that X € B. 


5.8 In the two-dimensional plane, assume nodes are mobile and each takes a roughly 
direct path through a cell. At a cell boundary, an independent and uniformly distributed 
random change of direction occurs for each mobile. Assuming the cells are roughly circular, 
a sample path of a single mobile is depicted in Figure 5.6, where the dot at the center of a 
cell is its base station. Also, we assume that the average velocities of a mobile through the 
cells are IID with density f(v) over [Umin, Vmax] [98]. The mobiles are initially distributed 
in the plane according to a spatial Poisson process with density ô nodes per unit area, recall 
Example 3.1. Finally, assume that the cells themselves are also distributed in the plane so 
that, at any given time, the total displacements of the mobiles are IID? and, consequently, 
the result of Problem 2.16 applies. 


(a) Find the mean rate Àm of mobiles crossing into a cell of diameter A. 


(b) How would the expression differ in (a) if velocity and direction through a cell were 
dependent? 


?The base stations could also be randomly placed according to a spatial Poisson process with density 6’ < 6 and 
the resulting circular cells approximate Voronoi sets about each of them, see Section 7.3.1. 
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Figure 5.6 A single mobile’s motion through circular cells. 


CHAPTER 6 





DYNAMIC ROUTING AND ROUTING WITH 
INCENTIVES 





In this chapter, we begin with a brief general overview of routing issues. After preliminary 
material on optimization is given, an introduction to routing with incentives is presented. 
A section briefly describing an approach to joint scheduling and routing in a multihop 
wireless context follows. We conclude this chapter with a section on traffic load balancing 
on multiple paths. 


6.1 GENERAL ROUTING ISSUES 


In Section 5.1 we assumed a loss network had a constant demand A, per route r, i.e., 
static routing. A Jackson network used purely randomized routing but with fixed (static) 
routing probabilities. Conversely, deployed routing algorithms in the Internet or in multihop 
wireless contexts are dynamic, i.e., responsive to changes in network state at all times. 
Also, routing algorithms in these contexts are highly distributed because network operating 
conditions potentially involve: 


e a large scale with respect to traffic volume or geography or both, and/or 


e high variability in the traffic volume both at packet and connection/call level on short 
time-scales (possibly due in part to the routing algorithm itself), and/or 


e potentially high variability in the network topology due to, for example. node mobility, 
channel conditions, or node or link removals because of faults or energy depletion. 
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Optimality results of routing protocols, specifically the Dijkstra and Bellman-Ford al- 
gorithms for unicast routing, are described in networking texts such as [139, 220] and in 
greater theoretical detail in [48, 206]; see Appendix A. These algorithms typically assume 
that costs (or “metrics") C, of paths/routes r are additive, i.e., 


oe eo (6.1) 


ler 


where c represents the cost of link 1. Such nonnegative link costs include Boolean hops, 
i.e., œ = 1 for all active links / (leading to path costs C, that are hop counts as used in the 
Internet), and those based on estimates of access delays at the transmitting node of the link. 
Alternatively, path costs could be based on the bottleneck link on the path, i.e., 


Cp, = maxcy. (6.2) 
ler 


Such link costs include those based on residual energy e), e.g., 


or an estimate of the lifetime of the transmitting node of the link [39]. More complex 
two-dimensional link metrics of the form (c”, c”) may be employed to consider more than 
one quantity simultaneously, delay and energy for example, or hop count and BGP policy 
domain factors. One can define (lexicographic) order 


(cii < (c3,c) (6.3) 
to mean 
eae or {c = c3 andc} < c3}. 
and define addition of link metrics (to obtain route costs from link costs) as 
(fd) + (S,ck) = (F +G, mafe, gh); (6.4) 


see Section 6.4.2 of [195] or Table 1 of [206]. For example, defining c” € {1, 00} in order 
to count hops of a path and basing c¥ on the residual energy of the transmitting node, the 
chosen paths will be those with the highest bottleneck energy among those with the shortest 
hop count to the destination. 

Alternatively, one can determine optimal paths according to one metric (the primary 
objective) and choose among these paths conditional on another metric (the secondary 
objective) being less than a threshold. For instance, suppose the primary objective is to 
minimize (bottleneck) energy costs and suppose a route r has C7 hops and C¥ energy cost 
according to (6.1) and (6.2), respectively. Now suppose a route r’ = r U {l} is being 
considered (by appending link / to r). The network will use the (6.4) rule for addition, i.e., 


(CZz,C%) = (cf +C?,max{c/,C¥}) 
if cf + CT < 0°” for some threshold 6* > 0. Otherwise it will set 
(CP, Ch) = (00,00) 


and, consequently, the network will not use route r’ nor any route r* that uses r’ (i.e., 
r’ C r*). Similarly, the network can find routes with minimal hop counts (primary objective) 
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while avoiding any link with energy cost c > 6” > 0 (ie., the residual energy of the 
transmitting node of the link e < 1/0”). 

A survey of routing protocols for (wireless) mobile ad hoc networks (MANETs), in 
particular Ad Hoc On-Demand Distance Vector (AODV) and Dynamic Source Routing 
(DSR), is given in [29]. On-demand source routing may be used when, for example, routes 
need very frequent updating due to node mobility or when forwarding decisions to all 
destinations is impractical to maintain at each node as may be in the case of a large network 
with a "flat" address space. Source routing can be implemented with a "threading" approach 
[71] or more simply as a limited-scope flood when feasible; see Section 7.2.2. 

The basic Internet routing protocols, such as the inter-autonomous-system (AS) BGP, 
intra-AS OSPF protocol, and Internet Group Management Protocol (IGMP multicasting), 
are described in [139, 172, 220]. A brief overfiew of the OSPF, BGP, and MPLS protocols 
is given in the following section. 


6.1.1 Discussion: IP forwarding 


Recall that for Internet (layer 3) routing, a higher layer message is segmented and encap- 
sulated into Internet packets by the transmitting end host. The IP packet headers contain a 
(destination) 32-bit IPv4 address of the receiving end host. If this address is not cached at 
the sender or if the message is not in response to another from the receiver, it may be deter- 
mined by the sender via the domain name resolution system (DNS) given the domain name 
of the receiver. Routers receiving packets make forwarding decisions based on the packets’ 
destination IP addresses. These decisions are determined in a scalably hierarchical fashion 
typically by the OSPF protocol within an autonomous system (intra-AS, i.e., within a pe- 
ripheral subnetwork) and by BGP inter-AS. Forwarding works by a longest prefix matching 
of an arriving packet’s destination IP address to the stored entries in a ternary content ad- 
dressable memory (CAM). The ternary CAM associates address prefixes to output links of 
the router. These associations are derived from the router’s routing information base (RIB). 

Under BGP, gateways "advertise" (notify peer routers under BGP) the ASes to which 
they are directly connected, i.e., for which they are the "origin" routers. More generally, 
BGP routers advertise to their neighboring peers their shortest (in hops) paths to different 
groups of ASes. Both gateways and interior routers will update their RIBs in response to 
such advertisements. Without policy considerations, BGP speaker z will forward packets 
destined to subnetwork S to z’s neighboring (next hop) BGP peer w that is [48] 


arg min Cz.) + Ciw,s); (6.5) 
where Cz w) is the hop cost of the link (z, w) between z and its neighboring BGP peer w, 


and C(w,s) is w’s current path cost to S as advertised to z. BGP’s use of path vectors, 
wherein the entire AS chain is advertised rather than just the path lengths, allows it to: 


e apply policy rules to any AS on the path, 


e check for loops, i.e., avoid the situation where the same AS appears two or more 
times on a single path, and thereby 


e avoid potentially slow "count to infinity” convergence problems [139]. 


Still, the scale of operation and distributed nature of BGP results in issues of convergence and 
reachability (graph connectedness). Poison reverse, dynamic minimum route advertisement 
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interval (MRAI) adjustments, and other devices to dampen the frequency of route updates 
trade off responsiveness (to, e.g., changing traffic conditions, link or node withdrawals) 
and stability/convergence properties. Finally, BGP advertisements (and the resulting RIBs 
and forwarding decisions) are made economical by aggregating those to ASes with similar 
routes, in particular those with common address prefixes (stubs). 

Within an AS, it is often feasible for routers to periodically flood the network with their 
link state information. So, each router can build a picture of the entire layer-3 AS graph from 
which loop-free optimal (minimal-hop-count) intra-AS paths can be found in the manner 
of Dijkstra-based OSPF protocols. A hierarchical OSPF framework can be employed on 
the component "areas" of a large AS. Under OSPF, router z will forward packets ultimately 
destined to router v according to the subpath p to a neighboring (predecessor) router rp of 
v that is 


arg min Cp + C(r,,v): (6.6) 
P 


where Cp is the path cost (hop count) of p. Dijkstra’s algorithm works iteratively by calcu- 
lating optimal paths with increasing distance from the router node at which the algorithm 
is executing [48]; see Appendix A. A fundamental property of global (network wide) solu- 
tions to (6.5) or (6.6) is that all subroutes of any optimal route are themselves optimal. This 
property, easily proved by contradiction, facilitates "connectionless" (i.e., IP) forwarding 
along optimal paths; again, see Appendix A. 

The 48-bit Ethernet (layer 2) media access control (MAC) addresses are not organized in 
a hierarchical manner like IP, i.e., Ethernet domains (within an AS) have a flat address space 
wherein proximal end systems may have numerically unrelated MAC addresses. This is 
workable since peripheral Ethernet clouds typically connect only on the order of thousands 
of proximal end systems, as compared to hundreds of thousands of different subnetworks 
(each designated by an IPv4 address prefix/stub) across the world connected together via 
IP. Routing in Ethernet works by flooding and self-learning mechanisms ([139], p. 471), 
where the latter is similar to reverse-path forwarding as described in Section 7.2.2. 

We will return to related issues in Chapter 7. 


6.1.2 Discussion: MPLS 


The operation of MPLS label-switched paths (LSPs) in the Internet, briefly discussed at the 
end of Section 4.5, is typically achieved via downstream-unsolicited label distribution and 
ordered control [153]. Under downstream-unsolicited label distribution: 


e labels are assigned (bound) to peripheral IP subnetwork address blocks and are "ad- 
vertised” upstream by egress LERs; 


e an upstream LSR simply binds the received labels to new ones in the label set it 
controls and, in turn, advertises the new ones further upstream toward ingress LERs; 
and, finally, 


è once ingress LERs are in receipt of labels from their downstream, neighboring LSRs, 
LSPs to IP subnetworks (connected to egress LERs) can be set up by specifying a 
FEC of packets that are to use them. 
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Basically, an LSP is requested by specifying a destination IP subnetwork (egress LER), 
ingress LER, and FEC. 

Under ordered control, the process of determining the intermediate LSRs and the set 
up of the labels that will correspond to a given LSP also begins at the egress LER. Labels 
are sequentially assigned along the LSP in the opposite direction of the flow of the FEC of 
data packets it will carry. This is similar to the "receiver-driven” approach of the resource 
reservation protocol (RSVP) {12, 239]. 

Alternatively, the Internet’s native packet-routing protocols could be used on a "pilot" 
setup packet to determine the actual path of the LSP in a manner forward from the ingress 
(source) LER of the LSP. This pilot packet may actually be the first data packet of the FEC 
so that there could be negligible LSP setup delay. In principle, MPLS can be used by an 
ISP in concert with BGP to set up a VPN connecting the sites of a customer [187]. 


6.2 UNCONSTRAINED OPTIMIZATION 


In order to be able to discuss revenue maximization in loss networks in the next section, 
we will now review basic gradient descent/ascent methods for local optimization. Consider 
a function f : R” — R to be minimized, i.e., points z* € R” at which f achieves its 
minimum value(s) are to be found. In the following, we will explain an iterative algorithm 
that will eventually determine a local minimum point of f. 


Lemma 6.2.1. If the gradient of f atx V f(x) 4 0, then any vector h such that AVE (x ya 
Oisadescentdirection of fat z, i.e., there exists a cea > Osuch that f(z+ah) < f(x) 
forall a < Qmax, Le, —V f(x) is a descent direction of f at z. 


Note that ATV f(a) is the inner (or "dot") product on of the vectors h and V f (£). We 
say x* is a local minimizer of f if there exists a ô > O such that 


f(z") < fz) 


for all z such that |z — z*| < 6. If this statement holds for 6 arbitrarily large, x* is said to 
be a global minimizer. If this statement holds with strict inequality (for x # x”), then x* 
is said to be a strict local minimizer. 

A function f is convex if its n x n Hessian matrix is positive definite, i.e., 


= > 0 
7 22 (= zy = DVR, os; = L (2) 
for all z € R” and y € R”\0; then a local minimizer is a global minimizer. Recall that an 


equivalent definition for convexity was given at the end of Section 1.6. 


Theorem 6.2.1. If x* is a local minimizer of f, then 


Vf) = 0. (6.7) 
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That is, (6.7) is a necessary condition for optimality. 


Theorem 6.2.2. x* is a local minimizer of f if 


Vi (z*) =0 and zaz”) > 0. (6.8) 


That is, (6.8) is a sufficient condition for optimality. 
The following algorithm, called steepest gradient descent, will converge from an arbitrary 
initial point 2) to a local minimum of f: 


0. Select initial zy € R” and set index 2 = 0. 
1. Compute search direction h; = —V f (x;). 


2. Compute the (real-valued) step size 


a, € A(z,) = argmin f(z; + ah,). 
a>0 


3. Update 


Ziyi = Lt ajhj. 


4. Increment index 7 — i+ 1 and go to step 1. 


So, the steepest gradient-descent algorithm generates a sequence {x;}%2, in R”. 
An accumulation point of a sequence is the limit of a convergent subsequence. See 
(28, 145, 173] for a proof of the following basic result. 


Theorem 6.2.3. Each accumulation point of the steepest gradient descent algorithm is a 
local minimizer of f. 


One can also formulate the steepest descent algorithm (with no "gradient" qualifier) by 
jointly optimizing over the step size and descent direction in step 2. In addition to requiring 
computation of the gradient of the objective function, steepest gradient descent solves 
our original unconstrained n-dimensional optimization problem (at least locally) with an 
iterative algorithm that requires the solution of a one-dimensional optimization problem at 
each step. The one-dimensional search for the step size required in step 2 of a steepest 
gradient-descent iteration can be made more practical by discretizing the search space; 
the resulting algorithm is called the Armijo gradient-descent method [173]. In practice, 
a gradient-descent iteration can be terminated when the change in f after an iteration, 
f(2;) — f(aj.1), becomes sufficiently small. When a function is not unimodal (having 
more than one local minimum), a better local minimum can perhaps be found by repeating 
a gradient-descent algorithm at a variety of different (say, randomly) chosen initial states 
to. 
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6.3 REVENUE MAXIMIZATION FOR LOSS NETWORKS 


In this section, we consider the simple problem of tuning a randomized routing policy to 
maximize a single, global (network wide) revenue objective (6.9) that explicitly depends on 
steady-state connection blocking rates. A general deficiency of such problem formulations 
is that they assume a centralized decision maker who knows the network state everywhere 
and at every given time. Specifically, we consider how randomized routing could be adjusted 
to maximize the revenue generated by a loss network. Let A,n,n be the intensity of the 
Poisson connection arrival process between end users n and m and let Ran.» be the set of 
distinct routes connecting them. Thus, 


y Ar = Anns 


rTERmn 


We assume that the mean holding times for all routes in Rm,n are the same, i.e., jir = 
lmn for all r E€ Rm.n. In this context, randomized routing determines the component 
route intensities À, of Am.n for each (unordered) pair of end systems m and n so that the 
connection arrival process to each route is independent and Poisson. As in the previous 
chapter, we will find it more notationally convenient to work with traffic load variables 
p = A/jethan with connection arrival rates À. 

Suppose that the revenue generated by a connection on route r 1s w, dollars per second 
(typically, routes connecting the same end systems will have the same rate of revenue 
generation). In this section, we assume that the goal of routing is to maximize network 
revenue as defined below. By Little’s formula and PASTA (refer back to Chapter 3), the 
average number of connections on route r is 


1 

rr (1 — B,{p, @)) = pr(1— B-(p.¢)). 
Te 

where we have explicitly shown the dependence of the blocking probability B, on the 

incident route loads p and the link capacities c; see Equation (5.4). Thus, the total mean 

rate of revenue generation in steady state is 


Wp.) = >  wrer(1— B-(p.0)). (6.9) 


With this real-valued objective function, we can study the performance of routing algo- 
rithms. 

Suppose the connection arrival rate Am.n has increased by a smal] amount so that the 
network must decide how to correspondingly increase the load on a route, or routes, con- 
necting end systems n and m. Further suppose that the route that will be chosen by the 
network will be the one that results in the greatest increase (or least decrease) in the network 
revenue rate W. The chosen route will therefore be the route r E€ Rm.n for which W has 
the largest sensitivity (OW/Op,) to pr. 


Theorem 6.3.1. 


Slee) = (1—B,(p.c)) (wr — [W (p.c) — W(p.¢— Ac,)]) - 
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Proof: First note that 


ow _ OBx(p,¢) 
T (p,c) = wr(1—B,(p,c)) - 2 Mel ag (6.10) 





By direct differentiation, 


dG(c) 














— = G(c— Ae,) (6.11) 
Opr 
and, therefore, 
OBr(p,c) _ _ G(e- Ae, —Ae,) | G(c— Ae, )G(c — Ae) 
Opr Gc) G?(¢) 

_ _Glc-Ae, —Ae,) Gle- Ae) | Gle- Ae, )G(c~ Ae) 
E G(c) G(c - Ae,) G?(c) 
z (1 — B,(p, ¢))[(1 — Br(p,c))-1 — B,(p,¢— Ae, )]. 

The proof concludes by substituting into (6.10). o 


To interpret Theorem 6.3.1, note that the increase in the revenue given by the sensitivity 
ƏW/ðpr has two components. The positive quantity (1 — B,(p,c))w, is the increase in 
revenue from an additional incident unit of load on route r, only. the fraction 1 — B, (p, c) 
of which will be admitted. That is, (1 — B, (p, c))wr is a "benefit." The negative quantity 


—(1~ B,(p,¢))[W(p,¢) — W (p, c- Ae, )] 


represents the lost revenue that this additional unit load on route r precipitates, i.e., the 
additional load will mean that more circuits on the links of route r will be occupied which 
will result in increased connection blocking on other routes that share links with r. This 
negative quantity is called a "shadow cost." 


6.4 CONSTRAINED OPTIMIZATION AND DUALITY 


We will next consider a kind of flow-based network like that of Section 5.2. To do so, 
some additional material in the theory of optimization is covered in this section. Consider 
a primal optimization problem with a set of m inequality constraints: Find 


arg min fo(z). (6.12) 
where the constrained domain of optimization is 
D = {xER"| fi(z) <0,Vi€ {1,2,...,m}}. 


For the example of a loss network, the constraints are 


filz) = (Az)-cq = yo Ly — Ch, (6.13) 


r | ler 
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where the index / corresponds to a link and m is the number of links in the network. 
To study (6.12), we define the corresponding Lagrangian function on R"7"": 


Lizu) = folz)+>_ vifi(z). (6.14) 
i=l 
where, by implication, the vector of Lagrange multipliers is v € R™. Let 
vu 2 0 


connote that all of the m Lagrange multipliers are nonnegative, i.e., v € (0.00). 


Theorem 6.4.1. 


Tak L(z,v) = win folz) = p*. 
Proof: 
ep, oe ni oC ifx ¢ D, 
max Leg) = í folz) if D, Ole) 
as desired. Oo 


So, by Theorem 6.4.1, we can maximize the Lagrangian in an unconstrained fashion to 
find the solution to the primal problem (6.12). 
Define the maximizing values of the Lagrange multipliers, 


u(x) = argmax L(x, v) 


and note that the complementary slackness conditions 
uf (2) fi(z) = 0 (6.16) 


hold for all z € D andi € {1,2,...,m}. That is, if there is slackness in the ith constraint, 
i.e., fi(x) < 0, then there is no slackness in the constraint of the corresponding Lagrange 
multiplier, i.e., v7 (x) = 0. Conversely, if f;(a) = 0, then the optimal value of the Lagrange 
multiplier v7 (x) is not relevant to the Lagrangian. Complementary slackness conditions 
lead to the Karush-Kuhn-Tucker necessary conditions for optimality of the primal solution 
[28, 145, 173]. 

Now define the dual function of the primal problem: 


gl) = min L(z,v). (6.17) 


LER” 


Note that g(v) may be infinite for some values of v and that g is always concave. 
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Theorem 6.4.2. Forall x € Dandv > 0, 


gv) < folz) 


Proof: For v > 0, 


gv) < L(z,v) < max L(z,v) = folz), 


where the last equality is (6.15) assuming z € D. o 


So, by Theorem 6.4.2, if we solve the dual problem, i.e., find 


i. i= max g(u), 


then we will have obtained a (hopefully good) lower bound to the primal problem, i.e., 
e 


Under certain conditions in this finite dimensional setting, in particular when the primal 
problem is convex and a strictly feasible solution exists, the duality gap 


p*—-d* = 0; (6.18) 


see [28]. To use duality to find p* and arg maxzep fo(z) in this case, suppose that a slow 
ascent method is used to maximize g, 


Un = UUn-1+0V9(U__1); (6.19) 


and between steps of the ascent method a fast descent method is used to evaluate g(v,,) by 
minimizing L(g, v, ), 


Ek = Zk- 42VaL(Ep1: Vn) (6.20) 


The process described by (6.19) and (6.20) is called an iterative subgradient method. Note 
that the step size a; may need to be chosen so that v,, > 0 for all n. 

Comprehensive treatments of optimization using gradient methods are given in [28, 145, 
173]. 


6.5 A DISTRIBUTED PRICING AND RESOURCE MANAGEMENT 
FRAMEWORK 


We now consider a kind of flow-based network like that of Section 5.2. Recall the intro- 
duction to constrained optimization and duality in Section 6.4 and consider a group of R 
users sharing a network consisting of m links (hopefully without cycles) connecting a pair 
of nodes. We identify a single fixed route r with each user, where, again, a route is simply 
a group of connected links. Thus, the user associated with each route could, in reality, be 
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an aggregation of many individual flows of smaller users. Each link / has a capacity of c 
bits per second and each user r transmits at x, bits per second. Link / charges «x dollars 
per second to a user transmitting x bits per second over it. 

Suppose that user r derives a certain benefit from transmission of x, bits per second 
on route r. The value of this benefit can be quantified as U, (xp) dollars per second. The 
user utility function U, is often assumed to have the following properties: U,.(0) = 0, U, 
is nondecreasing, and, for elastic traffic, U, is concave; recall Section 4.6. The concavity 
property is sometimes called a principle of diminishing returns or diminishing marginal 
utility. The nondecreasing and concave assumption may not be valid for individual users 
with applications requiring precise amounts of bandwidth [140]; in some cases, one can 
aggregate such inelastic demand applications into a call center which then may be modeled 
with an elastic utility [110]. 

Note that user r has net benefit (net utility) 


Ur (ar) — Br Y Ki 


ler 


Suppose that, as with the loss networks, the network wishes to select its prices x so as to 
optimize the total benefit derived by the users, i.e., the network wishes to maximize "social 
welfare," for example, 


R 
—folz) = > °U,-(2,) (6.21) 
r=1 
subject to the link capacity constraints 
filz) = (Av);-a < O forl<i<m. (6.22) 


We can therefore cast this problem in the primal form using the Lagrangian defined in 
(6.14), i.e., maximizing 


> U, (er) (6.23) 


is equivalent to minimizing fo(z). 

Since all of the individual utilities U, are assumed concave functions on R, fois convex on 
R”. Since the inequality constraints f; are all linear, the conditions of (6.18) are satisfied. 
So, we will now formulate a distributed solution to the dual problem in order solve the 
primal problem. First note that, because of convexity, a necessary and sufficient condition 
to minimize the Lagrangian L(x, v) over z (to evaluate the dual function g) is 


VzL(z* (v). v) = 0. 
For the problem under consideration, 


OL(z, v) f 
pg 


ler 
= —Uf(r-) + (Atv). 


168 DYNAMIC ROUTING AND ROUTING WITH INCENTIVES 


Therefore, for all 7, 


r;(v) 


(U)? (= n) 
ler 
= (U (ATY), (6.24) 


where the right-hand side of (6.24) is made unambiguous by the above assumptions on U». 
Assume that, at any given time, user r will act (select z+) so as to maximize their net 
benefit, i.e., select 


arg max U, (x -2% k = (UD! (= 3 : (6.25) 


ler ler 


By (6.24), the quantity in (6.25) is simply 7*(«), i.e., the prices x correspond to the Lagrange 
multipliers v. A user acting in this manner is sometimes said to be greedy. Therefore, by 
the statement of Equation (6.24) and recalling the dual function g of (6.17), 


glk) = L(a*(),4), (6.26) 


i.e., for fixed link costs k, the decentralized actions of greedy users minimize the Lagrangian 
and, thereby, evaluate the dual function. 

Following the framework of the dual algorithm (6.19) and (6.20), suppose now that the 
network slowly modifies its link prices to maximize g(x), where by "slowly" we mean that 
the greedy users are able to react to a new set of link prices well before they change again. 
To apply (6.19) as the method of modifying the link prices, we need to evaluate the gradient 
of g to obtain the ascent direction. Differentiating (6.26) we get 


Og(K 
LE [(Az"(s))e — a] 2 Ur (a7 (4) ne p? me >= an 


r| er 





il 





= (Az*(k)) =c; 
where the second equality is a result of (6.24). Thus, for each link L, (6.19) becomes 
(Kin = (Kt)n-1 + a1 ((Az* (kn) — 1) (6.27) 
or, in vector form, 
Kn = Bp- $+ a1(Az*(K, 1) e). (6.28) 


Note that the link price updates on (6.27) depend only on "local" information such as link 
capacity and price and link demand, (Az*(,,_,));, where the latter can be empirically 
evaluated. 

Suppose that we initially begin with very high prices «, so that demands x* (x) are very 
small. The action of (6.28) will be to lower prices and, correspondingly, increase demand. 
The prices will try to converge to a point «*, where supply c equals demand Az™ (k*). 

Note that in the present problem formulation the action of greedy users and that of the 
network attempting to maximize its revenue based on perceived supply and demand lev- 
els result in the maximization of social welfare fo. In [157], a passive framework was 
considered wherein round-trip delays, estimated by the users of TCP sessions, were used 
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instead of prices. Maximizing "social welfare" (6.23) may likely not be an objective of 
a practical system. Instead, a network may simply attempt to maximize its revenue by, 
say, controlling its price charged to end users for access, possibly subject to government 
regulations and to competitive networking market forces. Obviously, by selecting higher 
prices, it will lose some customers but make more money on those "premium” customers 
that remain (i.e., those that can afford the higher prices). A network’s additional desire to 
maximize utilization of its resources (or government regulation) may lead to service differ- 
entiation involving an affordable flat-rate best-effort service in addition to more dynamically 
usage-priced premium services. 


6.6 DISCUSSION: JOINT SCHEDULING AND ROUTING IN MULTIHOP 
WIRELESS NETWORKS 


Consider a network of V nodes with a set of L (one-hop) links and a set of F active end- 
to-end unidirectional, unicast flows. The rth flow requires an allocation of bandwidth that 
is proportional to the integer k, and uses unidirectional (one-hop) links / € r. Finally, 
suppose that the local medium access control is round-robin TDMA. 

Our objective in this section is to briefly describe the problem of formulating a schedule 
of round-robin link transmissions subject to the constraint that, at any given time slot, a 
node can either receive or transmit but not both, i.e., a node must avoid self-interference. 
By synchronizing local scheduling, this can be broken down to a bipartite graph-matching 
problem as discussed in Section 4.1.3. That is, in any given “global” time slot t, the N 
nodes are matched to themselves as specified by an N x N matrix M(t) of Boolean entries 
m,j(t), where 


mt = 1 if node ¿ transmits to j in time slot t, 
aes 0 otherwise. 


The duration of the time slots can be sized to account for transmission of acknowledgments 
by a receiving node upon correct reception of a data frame. A global schedule M of length 
T time slots is a set of T such schedules, i.e., 


M = {M(1),M(2),....M(T)}. 


The desired global schedule must meet the needs of the flows, of course. To this end, 
we can decompose each flow r € F into a set of k, - |r| one-hop unidirectional flows, i.e.. 
k» subflows for each link / € r. So, we require that 


T 


5 mia = 5 krl{ (ij er} 


t=1 rEF 


for all links / = (i. j), where 7 is the transmitting node and j is the receiving node of link 1 
in this notation. Naturally, the goal is to obtain the shortest possible schedule T satisfying 
this constraint. Note that the bandwidth allocation of a flow r will be roughly k,B/T, 
where B is the channel bandwidth. That is, each hop of r will have access to the channel 
the fraction k,./T of the time. 

In a multihop wireless setting there are additional constraints that need to be considered 
in the formulation of a schedule M. For example, certain pairs of links associated with 
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proximal nodes may interfere with each other and therefore cannot be scheduled in the same 
time slot. More specifically, if links (i, 7) and (a, b} interfere, then 


mij (t)Map(t) £ 1 


forall 1 < t < T. Note that this notion for a pairwise “interference” relation between links 
is more general than that considered in Section 4.1.3, i.e., two links need not necessarily 
have only transmitting nodes (e.g.,7, a) or receiving nodes (e.g., j, b) in common to interfere. 
Centralized and distributed cross-layer designs in different networking contexts, including 
those addressing versions of this scheduling problem, are described in, e.g., [65, 66, 91, 
132, 142] and in recent overviews [81, 144]. A typical but suboptimal approach would 
be to completely refresh the link layer schedule only periodically, i.e., as fast as existing 
computational resources will allow for such acomplex problem. A period between refreshes 
could have two phases: In the first phase, a quick and feasible "incremental" modification of 
the current schedule would occur to accommodate recently arrived and account for recently 
departed connections. In the second phase, no new connections are admitted and a refreshed 
schedule is computed based on the existing connections at the end of the first phase of the 
period. Clearly, connections arriving at the network during the second phase of a period 
may suffer significantly more setup delay. 


6.7 MULTIPATH LOAD BALANCING 


Reliability in the presence of faults and attacks is often achieved by proactively allocating 
redundant (backup) resources (e.g., bandwidth, computation, memory/state, energy) and 
possibly even setting up independent backup processes in their entirety [10]. For example, 
proactive setup, use, and maintenance of multiple paths between a source and destination 
for a single unicast session is required when issues of network reliability are important. 
When a path fails or is significantly disrupted, immediate switch-over to an alternative path 
is possible. This motivates the need for partial or total link or node disjointness of the paths. 
In an ad hoc wireless context, multipath setup is discussed in [16, 96, 108, 148, 167, 229]. 

Given multiple paths connecting a transmitting/source node to a receiving/destination 
node, the source can 


e transmit exclusively on one path, 


e load balance by dynamically selecting one path for each packet or consecutive group 
of packets to be transmitted (one can also envision this as a scheduling problem with 
multiple servers), or 


e transmit each packet on more than one path. 


Transmitting the same packet on multiple paths results in more reliable delivery of the 
packet at the cost of greater bandwidth resource expenditure of course. 

The focus of this section will be on load balancing based on the "biomimetic" notion 
of pheromone state, assuming that the performance (cost) of the links is not affected by 
the load balancing decisions under consideration. Moreover, the actual performance of the 
links need to be learned. Packets are interpreted as ants that associate pheromone with 
paths [8, 17, 34, 87]. More specifically, for a given pair of nodes, we explain a convergence 
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ants ants 





Figure 6.1 Simplified network topology. 


result [107, 233] resolving a path that is optimal in terms of throughput using an approach 
based on "regular" ants (messages) [213]. 

Consider two nodes connected with Z disjoint, bidirectional paths as in Figure 6.1. At 
one node, the total transmission rate is 1 — œ ants per unit time,! with 0 < a < 1, and the 
routing probabilities on the L paths are p = (pı, p2, .... pr), where ea = lite. 
p € Dp, the L-dimensional simplex. At the other node, the total transmission rate is a 
ants per unit time but the routing probabilities are q. Let 1 — y; be the probability that an 
ant on path 7 is dropped (irrespective of its direction of travel on the path) and assume that 
such transmission and dropping decisions are mutually independent. We further assume 
that there is no time decay mechanism for pheromone levels and that the path-dropping 
probabilities 1 — ~; are negligibly affected by changes in p or q, i.e., in the forwarding 
decision node by the source-destination pair shown in Figure 6.1. 

So, the joint process z = (p,q) can be modeled as a discrete-time Markov process on 
Ez x Nz [233]. The state transition 


(p.a) — (6'(p).q) (6.29) 


occurs with probability ag;7;, where 


EEC aoi 
l DC paN 
Ọj (p) = Cot i 
Cyl else, 
and a constant pheromone C > 0. Similarly, 
(p.a) > (pl) (6.30) 


occurs with probability (1 — a@)p;7;. Note that this process is time homogeneous. Let T 
be the time of the first transition (or "jump") of the Markov process, i.e., T is the smallest 


| Note that ants can be piggybacked on data packets. 
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positive time such that z(T) # x(0). Suppose that z(0) = (p,q) for some known initial 
set of routing probabilities and let (p,q) = E(x(T) | z(0) = (p, g)). Define 


L 
<py> = Ey = X pi: and <p, > = E(P) = X pa? 


Theorem 6.7.1. For any 0 < a < 1l and p,q, Y E€ =z, 


(1 -— a)Esy + aEgy = (1 — a)Epy + oEqy. 


Proof: We are given that z(0) = (p,q). First define 


z (1 — apii 


pt = Q&qiYi 
a= a)Epy + aEgy 


d * o iassa A 
ane i (1 — a)Epy + aEgy 
Correspondingly define the L-vectors p* and q*. The transition (6.29) of the jump process 
of the discrete-time Markov process occurs with probability q¥ and, similarly, (6.30) occurs 
with probability p;. By definition, 


L : L 
X o(p) f+ 2 eri 


p= 
i=l 
_ (1 apa + yaa c ds 
(1 -a)Epy+aEgy 7 @+1-" 
Therefore, 
C aEgy C 
Bea ek, in eee i EE 
BY ( C+1 oi] AANE rI 
i EAP SR AE. (6.31) 
C+1 (1—a)Epy + aEgy a1 Ag 
Similarly, 
C l-a 
Esy = Epy+ ? _EgyEpy). (6.32) 





. (E 
a BP. C+1 (1- a)Epy + aEgy ( py i ie 
Adding 1 — a times (6.31) to a times (6.32) and using the fact that 


Eny? + Egy? — 2EgyEpy 2 (Epy)* + (Eq)? — 2EqyEpy 


(Epy — Egy)” 
0 (6.33) 





IV 


proves the theorem. m 
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This theorem implies that the inner product of the vectors y and z(t), 


a 


Talt) = C= QE gy Pacey, 


is asubmartingale. Note that the first inequality of (6.33) is strict whenever ~y is not constant 
p almost surely and q almost surely. So, assuming that initially 47 x(0) is not zero on the 
subset of £z x i wherein eae is maximal, we can use the martingale convergence theorem 
2.9.1 to show that yT z(t) converges almost surely to a random variable that is constrained 
to this subset, i.e., to the optimal paths. See [233] for similar proofs of convergence of 
"uniform" and "synchronous" ant routing algorithms. 

In this section, we considered forwarding on multiple paths, where path costs had to be 
learned and did not depend on the forwarding decisions. This approach, based on positive 
reinforcement in the reverse directions, found the optimal paths for forwarding. To relate 
this problem formulation to that of Section 6.3, suppose that a total transmission rate X is 
to be divided among L unidirectional paths whose loss rates are assumed to be both known 
and load dependent. The transmission rate assigned to path 7 is x; > 0, where 


GE 
5 ri = X. (6.34) 
i=l 


Suppose that lost packets are selectively retransmitted so that in order to obtain an error-free 
throughput (or "goodput") of x; on path 7, the transmission rate into path 7 needs to be 


Ti 
—, (6.35 
1 — yilxi) 


where ¥;(z;) is the loss probability of path 7 such that +; is nondecreasing, 4;(0) = 0, and 
yi (2) — las z — oo. We can define a "soft" capacity of path i as, for example, 


— ,-1 
CG = % (0.05), 
i.e., the transmission rate at which the loss rate is 5%. Such capacities obviously depend on 


the existing load and available bandwidth especially at bottleneck nodes of the path. The 
objective now is to minimize the total transmission rate 


subject to (6.34). 
To further simplify matters, assume there is a single loss function model ~y such that 
4; (a) = y(a/a;) for all paths i, where we note that 


where c is the capacity when a = 1. Now consider the Lagrangian 


L 
G(x) +v (x = os «| . (6.36) 
{sl 
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where we note that the equality constraint (6.34) can be replaced in the problem formulation 
by the inequality constraint 


because G is increasing in all of its arguments. Defining 


ra) = $ (5g) 20 


it can be directly shown that the rates optimizing (6.36) are 
ce = a (v) (6.37) 


for all paths 7, where the Lagrange multiplier v is chosen so that (6.34) holds. Thus, 


r= 2 X= z X 
f ayi +agt +aL Cy +e tes: teL 





for all paths j, i.e., all paths are utilized (in proportion to capacity), not just those of maximal 
capacity. 


Problems 

6.1 Prove that the dual function g defined in (6.17) is always concave. 
6.2 Prove fo as defined in (6.21) is convex on R”. 

6.3 Prove (6.37). 


6.4 Consider a total demand of A between two network end systems having R disjoint 
routes connecting them. On route r, the service capacity is c, and the fraction of the demand 
applied to it is 7,, where "7, = 1. Find the routing decisions that minimize the average 
delay 


Da) = $ EET S 


where this expression is clearly derived from that of an M/M/1 queue. 


Note: This problem was extended to include the consideration of end-user dynamics in 
[134]. 


6.5 Consider the special case of the network of Section 6.5 that consists of just a single link 
l with capacity c so that the common charge per unit bandwidth on the link is s; = M, = M 
for all users r. 


(a) Show that the solution z maximizing (6.21) subject to (6.22) satisfies 


; =M ifz,>0, 
Uilen f <M ifz, =0. 
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(b) Now consider an iteration, where at the nth step each user rth requests an allocation 


n 


F a U. w 
WwW, = arg mas. r (a) — Ww 
(i.e., user r "bids" w? given price M”T!) and then the network sets its price at 


M° = Dane (6.38) 


and allocates z” = w"/M” to the users. Assuming there exists a unique solution 
M = M* to 


Yan) = e 


and that the corresponding x* > 0 for all r, show that M* is an equilibrium solution 
to the iteration above wherein (6.21) is maximized subject to (6.22). 


6.6 For the context of the previous problem, suppose each user is aware of the pricing 
strategy (6.38). So, each user r can infer the sum of the other users’ bids from their own: 


Wr = So wi 
ifr 
cM — wr 


(< — 1) Wr. 
Tr 


The following iterative bidding strategy can then be defined: The nth bid of the rth user is 


w 
we = arg max U, (= :) —w. 


(a) Show that the Nash equilibrium point w* of this game satisfies 


ijai e = M* ifat>0, 
Ul (xr) (1 T =) l < M* if x* = 0, 
where M* = 0, w*/cand 2* = w*/M*. 


(b) Letting M denote the equilibrium price for the scenario of Problem 6.5(b), show that 
M* <M. 


It was shown in [112] that such user knowledge of the pricing strategy will result in a 
reduction in social welfare $., U, (x+) of no more than 25%. 
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CHAPTER 7 





PEER-TO-PEER FILE SHARING WITH 
INCENTIVES 





Loosely interpreted, almost all systems involving a plurality of processes participating 
in a somewhat "symmetric" protocol can be interpreted in peer-to-peer (P2P) terms. An 
example of an asymmetric protocol is that governing client-server transactions wherein 
only the clients initiate transactions at any given time and the "always-on" servers respond 
to them. P2P systems have been used to implement name resolution services (e.g., for 
the Skype voice-over-IP system [18]), application layer multicast for content distribution 
[11, 82, 135, 143], and distributed computing [75]. P2P systems are also employed for 
management of Web caching [116]. An instance of a P2P content distribution network 
(CDN) could be a federated server farm that resolves queries for data objects (or "content") 
from the public Internet to a server that possesses the requested content; see, e.g., [212]. 
Also, "structured" P2P systems have been proposed to facilitate applications, such as mobile 
computing and multicasting, via the notion of address indirection [210, 218]. 

We will specifically focus on large-scale distributed P2P file-sharing networks involving 
peers that are not equally cooperative or trustworthy, though the discussion will be relevant to 
other applications of P2P frameworks. More general overviews of P2P networking systems 
are given in [6, 169, 191]. In this chapter, query resolution (also known as application layer 
routing, anycasting, or just search) will be discussed. Performance issues for unstructured 
and structured designs will be explored. The chapter is concluded by sections discussing 
incentives for cooperation in the downloading of data objects, focusing on cumulative 
reputation systems. 
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7.1 SUMMARY OF QUERY RESOLUTION 


In this section, we briefly summarize the performance issues that are common among file- 
swapping systems. A peer requesting a data object will launch a query among its fellow 
peers to identify peers that possess the object. As such, query resolution is seen to be 
performed at a protocol layer above that of TCP/IP, i.e., an "application layer” search. 
Once a query is resolved, a separate transaction typically occurs wherein the querying peer 
directly requests the object of interest from the identified "provider" peer and, if the provider 
agrees, downloads the object from (or swaps pieces of the object with) the provider peer. 
Both search and downloading phases will rely on the layer 3 routing underlay of the Internet. 
For file-sharing systems, performance metrics from a query’s point of view include: 


o query resolution latency in terms of time or forwarding hops; 


query loss probability, where loss could be due to TTL expiration or mistakenly 
forwarding to an inactive peer; 


the quantity or quality of the provider peers in the response, where quality could refer 
to willingness to upload, uplink bandwidth allocated for the upload, and other factors 
(like proximity [212]) that affect the upload performance; and 


o the quality of the obtained data object itself. 


Performance metrics also include measures of the resources consumed by the individual 
peers to resolve queries, i.e., 


e forwarding load and 
e acquisition and maintenance of related information. 


Corresponding system-wide measures (totals or averages) can also be considered. A mea- 
sure of "fairness" could be the variation in query resolution overhead experienced by the 
peers, possibly normalized by the quantity of their own requests. 

In addition, file-sharing systems need to be robust in the face of problems such as: 


e peer churn (nodes frequently leaving and joining), 


e the spread of false and potentially harmful data objects (spam and other content 
pollution, or malware), 


è the spread of false routing or name resolution information, 
è peers employing false or multiple identities, and 
è peers otherwise trying to undermine the systems in place to promote cooperation. 


Sometimes these problems are due to faulty peers. Sometimes they are deliberately caused 
by selfish or malicious peers. Defense against such activities may require significant over- 
head of the peers. For example, peers may be compelled to announce their departure 
beforehand and, if this rule is not well observed, a proactive “heartbeat” or periodic ping- 
ing system could be used to detect departed peers more quickly than a system based on 
observations of routing failures due to their departures. Some of these problems are further 
discussed later in the chapter, Section 7.5.3 in particular. 
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7.2 UNSTRUCTURED QUERY RESOLUTION 


We first discuss approaches to "unstructured" query resolution that differ in the degree to 
which they operate in a decentralized fashion. 


7.2.1 A centralized approach to search 


Napster performed query resolution with the aid of centralized content registries residing 
on Napster-owned servers that kept track of which peers currently held what data objects. 
The search mechanism of the Napster client contacted a centralized registry and the registry 
responded with a list of peers that likely possessed the desired data object. 

Under BitTorrent [23], the group of peers that are actively uploading and downloading a 
given data object are known as a "swarm" (for that data object). Peers search the Web and 
download a ".torrent" file corresponding to the data object of interest. The peer can then 
connect to a "tracker" that informs it of other peers currently participating in the swarm for 
the data object. Decentralized (trackerless) versions of BitTorrent employ a structured P2P 
system; see Section 7.3.4. 


7.2.2 A decentralized approach to search: Limited-scope flooding and 
reverse-path forwarding 


Gnutella used a decentralized approach involving limited-scope flooding. Essentially, each 
query was initially sent by the source (original querying) peer to a small number of peers 
(in this case just seven), i.e., a breadth limit to the flooding tree. If a queried peer did not 
have the data object and was not previously queried by a particular limited-scope flooding 
process, it in turn forwarded the query to seven more peers for that process. A single query 
thread was prevented from being forwarded more than 10 times by a TTL counter carried 
along each query message that was decremented each time the query was relayed, i.e., depth 
limits to the flooding tree. A single flood identifier, that may be randomly chosen by the 
source, is carried on every query packet of a given flood and enables peers to easily detect 
reception of the same query forwarded by different peers. 

If a queried peer possessed the desired data object, it would communicate back to the 
original querying peer via reverse-path forwarding [54]. Under reverse-path forwarding, a 
query is forwarded but not before the relaying peer appends its identifier to the query. Once 
resolved, a query response travels back along the path of peers that previously relayed it so 
that they can all learn about the data object’s provider peers. 

Gnutella’s performance was improved through the actions of "superpeers" (or "supern- 
odes"), i.e., cooperative, long-lived peers typically with significant resources (memory, 
computation and network access bandwidth) so that they were able to effectively handle 
very high amounts of query resolution traffic. One may interpret a P2P system that heavily 
relies on superpeers as only partially decentralized. Again, once the querying peer has iden- 
tified peer(s) that possess its desired data object, it then contacts them directly to request 
download of the data object. 

Freenet (47, 237] is a P2P system that employs limited-scope flooding and reverse-path 
forwarding to find and distribute data objects. That is, the data object itself can be sent 
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along the reverse path of the query for it, and is potentially cached by intermediate Freenet 
nodes, thereby obfuscating the peer that originally uploaded it, and thus protecting the 
privacy of (anonymizing) transactions especially with respect to entities outside the Freenet 
community. Also, the query originator may obfuscate his or her identity by varying the 
initial value of the query message’s TTL. Peers are connected so as to create a "small world” 
effect in order to reduce search time; see the end of Section 7.3.3. 

When items are cached, the age, size, and possibly frequency at which the item changes 
(i.e., is updated) factor into caching strategies. The age of a item in the cache can refer to 
the last time it was accessed. Clearly, to improve system performance, peers will prefer to 
cache the items that are popular (frequently requested and, hence, younger"), small in size 
(so that they do not consume significant memory and bandwidth resources), and largely 
static (do not require frequent updating to maintain consistency in the distributed cache). 
In Freenet, the data object itself may be the subject of caching and, in this case, the size 
of the item cached will be more of an issue than for the cases where only the identity of 
peer(s) that are providers for certain data objects are cached. Mechanisms for updating or 
discarding stale items in a cache may rely on set lifetimes or explicit notifications of item 
expiration in whole or part. When the cache is full, a "least recently used” rule may be 
employed to decide which cache entry to overwrite, i.e., overwrite the oldest entry. Again, 
this rule may be modified to consider the sizes and mean and variance of interupdate times 
of the affected entries [190]. 


7.2.3 A partially centralized approach to search 


The popular file-sharing system KaZaA employs a partially centralized query resolution 
strategy. That is, certain peers become superpeers each handling a group of (nonsuper) 
peers that are assigned to it. The superpeers are aware of the IP addresses and content of 
all of its assigned peers. Also, superpeers are aware of the identity of other superpeers in 
order to quickly resolve queries for data objects that its peer group does not possess. 

The voice-over-IP system Skype involves a similar group of superpeers for name resolu- 
tion, i.e., mapping the Skype login name of the desired recipient of a call to the corresponding 
end system’s IP address. In addition, a special Skype server interacts with client peers for 
login, public-key encryption, and possible billing purposes [18]. Once the destination end 
system of a call is identified, packets containing encrypted (for privacy) and encoded voice 
are transmitted between the peers over the commodity Internet. 

In the GIA system [40], a topology adaptation framework is used to "bias" the random 
walk to superpeers and thereby significantly reduce the overhead as compared to the limited- 
scope flooding approach of Gnutella. A peer estimates the query-processing rates of other 
peers and tries to associate itself with peers that are no slower than itself, i.e., it avoids 
sending to and relaying queries from peers that are relatively slower. This system creates 
a hierarchical query-forwarding topology in which peers of similar query processing rates 
are grouped. Also, GIA employs a tit-for-tat token system to reward cooperative peers; 
that is, tokens are accrued by a peer when it forwards relays and tokens can thereafter be 
redeemed by that peer when she or he wishes to launch queries of their own. 
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7.2.4 An example of search by random walk 


Consider a query resolution system that involves reverse-path forwarding. In particular, 
consider a series of queries, assumed not overlapping in time, from a peer population w that 
are launched for a single piece of information (say the identity of any member of a swarm 
of peers that are actively acquiring and disseminating a certain data object) that is known 
by a population f. For simplicity of the following model, suppose for each peer that each 
entry in its forwarding table (of other known peers) experiences a lifetime with mean 


1 

= (7.1) 

H 
irrespective of the frequency at which the entry was accessed or the time of the last access. 
Peers join the system via a randomly chosen access point in the f + w population. In 
addition, define: 


e is the total external arrival rate of new peers to the system (specifically to the f + w 
population), 


e q < À is the per-node rate at which queries are spawned from the population w for 
the swarm under consideration, and 


e ò is the per-node rate at which peers leave the system, i.e., 1/6 is the mean lifetime 
of a peer. 


Finally, suppose that search is conducted in a distributed fashion via a single memoryless 
random walk through the forwarding tables of the peers and that, at each step of this 
search, any remaining peer is equally likely to be chosen. Such a single-thread search may 
require less overhead than a limited-scope flooding approach but will result in longer query 
resolution times. 
Given f and w, the mean number of hops to resolve the query is 
f w-1 f 1 

uciti alee a-is] "p 


wl 


h(w, f) = 


Note that h(w, f) is equal to the mean number of w-population peers visited including 
the querying peer. By reverse-path forwarding, on average h(w, f) peers will leave the w 
population and join the f population. We further simplify the model by assuming peers 
that have launched a query, joined the f population as a result, and then reverted back to 
the w population may thereafter launch the same query again. The following deterministic, 
continuous ("fluid") mode] summarizes the dynamics of the system: 


dw w 


ae ner: — qh(w, f) — ôw + uf, (7.2) 
df | f. ep 
FE tahtu f) -8f -nf (7.3) 


Note that with regard to the exogenous arrival rate (À) terms, f /(w + f) is the probability 
that the entry point is in the f population. This system is depicted in Figure 7.1. Specifying 
a stochastic Markovian version of this model is left as an exercise. In [191], a scenario 
is described where the fraction of peers that can resolve the query is proportional to the 
square root of the querying rate, i.e., f/(f +w) x q; see Problem 7.1. In practice, a 
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Figure 7.1 Randomized search with reverse-path forwarding. 


swarm may experience an exponentially increasing rather than constant (A) arrival rate of 
new peers in the initial stages of its lifetime, followed by a decline in arrivals as interest 
dwindles in the data object that is the subject of the swarm. 


7.3 STRUCTURED QUERY RESOLUTION 


Structured systems assign data objects and peers to points in a discrete metric space. The 
common (standard) name used for any given data object, say 2, is mapped by a hash func- 
tion/table h to an element of a "key space", h(i). Note that a name resolution process 
may be required to discover the common name of a data object. Each peer is assigned a 
segment of the key space, nearby its own assigned key or keys, and is directly responsible 
for resolving queries of all data objects that map to an element of their segment. 

Structured P2P systems employ "consistent" [116] distributed hashing tables (DHTs) to 
distribute the load required for query resolution among the participating peers in a balanced 
way and to facilitate peers leaving and joining. Under consistent hashing, peers leaving and 
joining do not significantly affect the keys assigned to data objects and to the remaining 
peers so that the overhead associated with peer node churn is localized. In some consistent 
DHT systems, each key is 160 bits (40 hexadecimal digits) long so that key space {0, 1}1°° 
has 21680 elements. When each peer is assigned a single key, it is possible that the segments 
assigned to peers will significantly vary in size; to reduce any resulting unfairness, multiple 
keys can be assigned to each peer [117] so that each peer’s assigned portion of key space is 
a union of disjoint segments about its assigned keys (i.e., the peer’s "virtual nodes" in key 
space); see Problem 7.3. 
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When the data object of a query is not in the segment(s) of a queried peer, it should know 
of a peer that does know (or is more likely to know) and forward the query accordingly. 
Some of this information is acquired directly upon initialization of the peer. That is, with 
the aid of existing peers or by special-purpose servers, a newly arrived peer is set up 
with an appropriate forwarding (or finger") table of keys and IP addresses of other peers 
in the system. Once established, a peer can periodically query other peers to update such 
information and to ensure that peers in its forwarding table have not left the system. Also, the 
identities/keys of other active peers can be acquired indirectly by observing specific queries 
and their responses, both of which may be labeled by the addresses of the originating and/or 
forwarding peers under reverse-path forwarding. Observed problems with queries (say 
timeouts in waiting for responses to arrive via reverse-path forwarding) may be indicative 
of stale forwarding table entries. A peer can emit "probe" queries with destination keys 
close to its own simply in order to identify neighboring peers in key space from the response. 
A peer with multiple keys assigned may elect to use only one of them for forwarding queries 
[117]. 

There are many possible strategies for the management of both tables of data objects and 
tables of fellow peers for forwarding purposes. Clearly, entries that are known to be stale 
(e.g., peers that are known to have departed the system) should be removed from forwarding 
tables. Typical trade-offs are in play in the sizing of such caches: Larger forwarding tables 
will result in fewer querying hops/relaying but will require more memory, maintenance 
overhead, longer look-up times per node, and possibly more overhead in setting up new 
peers. 

In summary, from a data object’s point of view there are peers: 


e that possess the object, 
e that do not possess the object but know about peers that do, and 


e that neither possess the object nor know about peers that do and hence must forward 
a query for that object to another peer. 


From a peer’s point of view there are: 
e data objects it possesses, 


e atable (or "registry") of data objects nearby in the key space together with the identities 
of known peers that possess them, and 


e a forwarding table of other known peers. 


In the following, we explore structured P2P query resolution for a specific example that 
is related to those described in [143, 160]. The section concludes with a discussion of 
the distance metrics used by Chord [89, 211] and Kademlia [152]. For additional reading, 
see the discussion given in [216] comparing Chord, Kademlia and continuous-discrete 
[160]. Other well-known structured P2P systems are described in the general references 
[169, 6, 191], in particular Pastry [196] and Tapestry [240]. 


7.3.1 A structured P2P framework using Voronoi cells 


For the purposes of discussion, suppose a data object has: 
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e at least one provider peer that possess the object (and can provide it to others). 
Also, for query resolution, suppose a data object has: 


e atleast one primary peer whose identity is, by some chosen metric, typically closest to 
the object in key space (more than one primary is possible in the case where distances 
are tied) and 


e several possible secondary peers whose identities are closest to the primary peer(s) 
in key space. 


Both the primary and secondary peers (collectively called "index" peers) of an object are 
aware of one or more of its provider peers. Each peer therefore has a table of nearby data 
objects (i.e., for which they are either the primary or a secondary) and one or more provider 
peers for each object. 

The secondary peers offer an amount of redundancy that is required to facilitate man- 
agement of peer departures, especially when departures are abrupt due to, for example, a 
fault. That is, when a peer node departs, its closest neighbors in key space can immediately 
take over its query resolution responsibilities, assuming the departure is detected in a timely 
fashion. In addition to reliability, secondary peers of a data object may elect to respond 
to queries for it, thereby improving search performance (during times of high demand for 
the data object in particular). This rule can be extended to allow popular but distant data 
objects to be stored in secondary registries. 

To join a peer group, an end system could contact an existing peer, superpeer, or server to 
obtain the client application and the key it will subsequently use. Peer keys can be assigned: 


e close in key space to that of the peer assisting in setup, 


e with the IP address, public encryption key, or physical location of the joining peer in 
mind, 


e by explicit consideration of load balancing across the key space, or 
e by sampling the key space at random (which may also have load-balancing benefits). 


In this process, the new peer’s IP address and key are initially known to at least one other 
peer and can be disseminated by that peer to others, particularly those that are close to 
the new peer in key space. Given the identities of its neighboring peers in key space, a 
newly set-up peer can query them for their tables of data objects and peers, in particular to 
determine the objects to which it is closest in key space and primarily responsible for query 
resolution [143]. Implicitly in the following, when we state that a primary or secondary 
peer is aware of an object, we mean that it is aware of providers of that object. 


7.3.2 Specific approaches using Voronoi cells 
Suppose that hashing maps both peer and object names to identifiers in a key space 
K c ge 


where S¢ is the surface of a (d > 2)-dimensional hypersphere, i.e., K is a finite, (d — 1)- 
dimensional set. Let 
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o X CK be the set of peer identifiers and 
e Q C K be the set of data object identifiers. 


Note that the number of bits required to codify peer and object keys will need to be large 
enough to accommodate the peers and data objects. This is, of course, not a problem when 
the size of the key space is 216°, as in the previously mentioned example (wherein the key 
space can be interpreted as belonging to a circle by mod 2!®° arithmetic). Indeed, it is likely 
that a 160-bit key space will be very sparsely populated with data objects and peers. 

The objects A, of primary responsibility for peer x are those closest to it in key space, 
Les 


A, = {wERQ]|zr= argmin |y- w|}. 
yEX 


Thus, the convex Voronoi cells A, partition Q, i.e., they are mutually disjoint and 


LJ 4 = Q. 


LEX 


Let 
N, C X 


be the nearest neighbors of peer x € K, i.e., y € Nz if and only if Ay and A, touch (share 
portions of their boundaries). Note that in two dimensions |.V,| is simply the number of 
edges (or vertices) of the polygon (Voronoi cell) Az. Define the content "view" of a peer x 
as the combination of its primary and secondary sets of data objects: 


yENz 


That is, each peer x maintains knowledge of peers that possess all objects in V,. In the 
two-dimensional key space example of Figure 7.2, the first-order neighboring peers of the 
center (black) peer are shaded and all of their cells are completely depicted. Clearly, for 
a one-dimensional key space, |N,| = 2. For a two-dimensional key space. E|.V,,| = 6; 
see [163]. That is, with regard to the definition of secondary peers and local forwarding 
decisions, additional dimensionality in the key space obviously increases the number of 
one-hop neighbors. Additional redundancy may be achieved by expanding the secondary 
assignments to include neighbors of neighbors (i.e., second-order neighbors), or third-order 
neighbors, etc. For the case of second-order neighbors, 


V = U U Az. (7.4) 


YEN; ZEN; 


Again, a peer may include in its table the keys of distant but popular data objects so as to 
improve the performance of the system (note the concept of "unidirectionality” described 
in Section 7.3.4 below). Such information could be obtained by reverse-path forwarding 
of queries, recall Section 7.2. That is, a portion of the secondary data object registry could 
be employed in "most recently seen" mode to deal with flash crowds for data objects, 
irrespective of their distance to the peer. 

When a peer joins the network, a protocol among it and its neighbors in key space 
will result in the creation of its primary and secondary lists of objects and corresponding 
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Figure 7.2 Two-dimensional Voronoi cells with N; = {y1,..., ys}. 


modification of those of its neighbors [143, 160]. Construction of Voronoi diagrams, and the 
"dual" Delaunay graphs having links between all pairs of neighboring peers, is described 
in, e.g., [86, 143]. Upon departure, all of the peers whose “views” contained objects 
that primarily belonged to x must adjust their object lists. In particular, the immediate 
(first-order) neighbors of x must negotiate to adjust their primary list of objects. This is 
graphically depicted for the two-dimensional case in Figure 7.3. 

To describe a local forwarding rule for this system, suppose that a peer with key x 
desires an object with key w. So, x needs to find a primary or secondary peer of w. Query 
resolution can operate so as to select the known neighboring peer y of x whose identifying 
key is closest to that of the data object w. Assuming "closest" is determined by simple 
arithmetic (Euclidean) distance and ignoring the boundary effects of modulo arithmetic, 
one can determine forwarding decisions by performing a longest prefix match of the key of 
the requested data object against the forwarding table of peer keys. 

If each forwarding step is to a peer that has at least one more (most) significant bit in 
common with that of w, then the maximum number of hops for any query will be bounded 
by the minimum of the number of peers and the size of the address space in bits. In the 
system proposed in [143], mean query latencies (network diameters) that were of the order 


of \/|X|/4 hops were observed. 
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5 (b) 


Figure 7.3 Two-dimensional Voronoi cells of (first-order) neighbors with and without the peer at 
the center. 


In the CAN system [178], a joining peer is assigned an address in key space z selected at 
random. The new peer then discovers the identity of the existing peer x that is the primary 
for its assigned address, i.e., z, x£ E€ Az, say by querying a data object with key equal to its 
own address. The set A, is then partitioned into two sets A’, and A’, so that 


e all primary sets are hypercubes aligned parallel to the axes of key space and 


e x and z are reassigned to addresses x’ and 2’ inside their respective primary sets, i.e., 
z € A, = Ay anda’ € Al = Ax. 


The process is reversed (primaries are merged) when a peer leaves. Clearly, if the new 
addresses x’, z’ are chosen at the center of their primary sets, then the primaries are Voronoi 
sets generated by the addresses of the peers in key space. 


7.3.3 Variations in the design of search, including Chord 


We have thus far described an approach that is similar to a distance vector method in that 
queries are forwarded to neighboring nodes in key space based on information that is local 
to the forwarding peer in key space, noting that, in application layer routing, neighboring 
peers in key space are not necessarily physically proximal (as they would be in a "spatial" 
graph), i.e., we are assuming just a "relational" graph connecting peers in key space. Also 
note that the primary table of data objects A, could instead correspond to a fixed region 
about x, thereby requiring more careful assignment of key space identifier(s) of peers so that 
the key space is completely covered by the A, and so that there is adequate redundancy. 
If the number of objects |Q| and peers |X| are large, the former may be achieved with 
high probability by a uniform "randomness" of the hash function assigning keys to data 
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objects; again see Problem 7.3. One can envision a highly decentralized and dynamic 
“opportunistic” system in which peers may cache the identities of any distant (in the key 
space) peers, information that may be obtained through reverse-path forwarding of queries. 
Typically in a highly decentralized system, it is also clearly of interest to include mechanisms 
to avoid loops in forwarding paths. As in IP routing, a TTL mechanism could be used to 
abort queries that have been excessively (unsuccessfully) forwarded. 

In this section, "long-distance" entries in forwarding tables will be considered. Assume 
a one-dimensional 160-bit key space and suppose that every peer x € X: 


e is a primary peer for D = |A,| data objects and 
e is aware of K other peers for forwarding purposes, i.e., forwarding tables of size K. 
That is, the peer x is primary for all data objects with keys that have 
160 — log, D 


most significant bits in common with z (again, ignoring boundary effects when x ~ 2160), 
Further suppose that the kth entry of its forwarding table is a peer z in the interval 


(2 + 2171D) mod 21©, (x + 2*D — 1) mod 216°], (7.5) 
where 
2* D = 160, 


i.e., we logarithmically partition the key space about x and assume the peer is aware of 
another in each of the A elements of the partition, as in the manner of Chord [211] (wherein 
z is chosen as the smallest known peer identifier in its interval). Again, queries for data 
objects y are relayed to peers whose keys are closest to y (mod 2!©°) among those stored 
in the forwarding table. For this system, the following result follows by simply showing 
that each forwarding step results in halving the distance between the peer’s address and the 
data object every time the query is forwarded [211]. 


Theorem 7.3.1. Every query under Chord will be resolved within K hops (relays), where 
K is the size of the forwarding tables of the peers. 


E EXAMPLE 7.1 


Suppose a query for a data object with key y is launched by peer xg. The case where 
D = 2'°© and K = 4 is depicted in Figure 7.4, i.e., the primary domain of a peer, 
indicated as a bold line segment in the figure, is greatly exaggerated and the size of 
the forwarding table is chosen small for the purposes of illustration. The partition 
of the key space of the forwarding table of peers xo and x, are depicted in Figures 
7.4(a) and (b), respectively. Key space is typically depicted as a circle (the bottom 
of Figure 7.4) to reflect modulo arithmetic operations on it. Note that each of the 
forwarding decisions, e.g., tọ — xı and x; — £2, result in at least halving the 
distance between the current peer handling the query and the data object, i.e., the 
progress of the distance is such that 


ly—winil < $ly—-ail, 
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xo + 21° = zo mod 219° 
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Figure 7.4 A 160-bit key space partition of (a) peer zo and (b) peer x1. 


where z; is the ith peer to relay the query. Also note how search is conducted only 
in the clockwise direction in Figure 7.4; see Problem 7.5. 


A point here is that (one-hop) forwarding decisions are not necessarily to neighboring 
peers. That is, in this approach, each peer needs to be aware of a larger community of peers 
so that not only nearby peers in key space are eligible for forwarding decisions. Hybrid 
approaches involving both local and nonlocal forwarding have been proposed, e.g., Pastry 
[196], hierarchical search structures [191], and small-world graphs [222, 103]. In one of 
the networks of [131], key space is a two-dimensional lattice with the /; metric 


d(x,y) = |z -y| + |z2— yal, 


peers are aware of their immediate neighbors, and peers have just one long-distance con- 
nection to a peer chosen at random. In this setting, a query process itself is a Markov chain 
and the proof of querying performance in [131] involves a logarithmic partition of key space 
(about the destination y) that is similar to the "deterministic" finger table structure later used 
by Chord for the peers. 
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Non-local forwarding potentially reduces search delays but may have disadvantages 
when handling peer arrivals and departures (churn). Recall that under the "Voronoi" ap- 
proach described earlier, peers in the forwarding table can be polled to recover from the 
departure of a neighboring peer. Again, a peer can react to a detected departed entry in its 
forwarding table by querying for a data object (possibly a fictitious one) whose key is close 
to that of a desired peer in key space and rely on reverse-path forwarding information in the 
response, or by simply querying known long-lived superpeers (152, 218]. Alternatively, 
peers may proactively identify multiple peers in each element of their partition of key space 
under Chord as mentioned at the start of this section. Clearly, for many different contexts 
both structured and not, given a critical mass of densely connected superpeers, routing is 
facilitated by simply forwarding the query to the nearest superpeer and letting them do most 
of the work. 


7.3.4 The Kademlia example 


In this section, we briefly discuss the Kademlia structured P2P file-sharing system [152], 
on which a recent "trackerless" version of the popular file-swapping system BitTorrent is 
based. Under Kademlia, data objects and peers are assigned to elements of an n-bit key 
space. Kademlia uses a bitwise exclusive OR (XOR) distance d between points in key 
space: 


d(z,y) = zy (7.6) 


for all x,y € {0,1}”. For example, if n = 5, x = 01101 (13 in decimal), and y = 10100 
(20 in decimal), then 


d(01101, 10100) = 11001 (25 in decimal). 


It is left as an exercise to show that d satisfies the formal properties of a metric on the key 
space, i.e., that 


d(x,y) > 0 foralls # y€ {0,1}", 

d(x,y) = 0 & r=y, 

d(x,y) d(y,x) forall x,y € {0,1}”, 
d(x,y) +d(y,z) > d(z,z) forall x,y,z € {0,1}", 


where the second-to-last property is symmetry and the last property is the triangle inequality. 

Suppose the peer with key x; receives a query for a data object with key y. Moreover, 
suppose that peer x, is not aware of a provider for y and so must relay the query. Peer 
Tk will search its forwarding table for the identity of another peer x+: that is closest to y 
according to the XOR metric so that d(z,, y) > d(£k+1,; y). Note that this metric benefits 
from a "unidirectional" property, as does that of Chord, i.e., given a key y and distance 
A > 0, there is a unique solution z to 


d(a,y) = A. (7.7) 


Under Kademlia, a peer node becomes aware of such information by queries for content each 
of which is assumed to be labeled with the sending peer’s key and responses that also have 
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the recipient’s key, as in reverse-path forwarding. Also, a peer node can perform remote 
procedure calls (RPCs) to ask its fellow peers for such information and can ping peers to 
check whether they are still "on" the network [152]. Peer nodes with long-term presence, 
i.e., those with well-developed lists, are important to such systems. The distribution of 
the lifetimes L of peers of file-sharing networks was empirically observed to satisfy the 
following property [198]: 


f(t) = P(L>t+60|L>t) 


is nondecreasing and f(100 minutes) ~ 0.8. For this reason, Kademlia does not delete 
peer nodes from a ("K-bucket") forwarding list even if they have not been been recently 
observed participating in a query or response, i.e., "stale" peers are potentially removed 
only when the list overflows. Similar reliance on powerful and key space knowledgeable 
superpeers has been proposed for other systems, e.g., a kind of hierarchical version of 
Chord called Iridium [170]. It may be feasible that small communities of superpeers may 
continually exchange significant amounts of up-to-date key space information, i.e., tables 
of data objects and keys of active "regular" peers. 

Note that the unidirectionality of the metric d implies that all queries for a given data ob- 
ject will converge along the same paths to it. So, peers along those paths that cache look-ups 
(again facilitated by reverse-path forwarding) may alleviate hot spots. Also note that resis- 
tance to the distribution of phony data objects can be achieved in a decentralized fashion by 
downloading the same content from more than one peer, an objective that unidirectionality 
may make difficult to achieve. One simple work-around is to respond to a query and, in 
addition, continue to forward the query but with an indication in it that a response has been 
previously issued, together with the identity of the providing peer given in that response, to 
ensure that a different providing peer can be given in subsequent responses. In this context, 
reverse path forwarding may not necessarily be in effect for all queries. Finally, note that 
multidimensional key spaces tend to have path diversity, instead of unidirectionality, which 
may be valuable for load-balancing purposes, especially in the presence of a flash crowds 
of high demand for certain data objects. 


7.3.5 Discussion: Spatial neighbor-to-neighbor graphs 


Note that thus far we have not necessarily associated the key space identifier of a peer 
with that of its IP address or physical location, i.e., "proximity" routing. Recall that in 
the Internet, BGP economically forwards packets based on address prefixes (stubs) that 
correspond to a group of subnetworks that lie in the same direction (output link) from a 
given router’s point of view. In terms of latency and bandwidth consumption performance, 
such an association would be beneficial for, for example: 


èe multicasting in which physically proximal peers are also proximal in the multicast 
tree [11, 43, 82, 135, 143], 


e individual downloads from decentralized content distribution networks (1.e., resolving 
queries for data objects to servers that are physically close to the querying peers) [212], 
and 


e route-to-location services of a sensor network (e.g., to resolve a query for a temper- 
ature reading at a certain location) [59]. 
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On the other hand, by leveraging superpeers one may be able to feasibly perform large- 
scale routing in "flat" address spaces; see, e.g., [32, 218]. Also, there may be reliability 
disadvantages of high correlation of peer keys with physical location; see the next section. 
A hybrid relational/spatial addressing system is also possible using a two-dimensional key 
with only one component having spatial properties; forwarding could be performed by a 
lexicographic rule as discussed in Chapter 6 at Equation (6.3). 


7.4 DISCUSSION: SECURITY ISSUES 


In this section, we will give an overview a variety of security issues that P2P overlay systems 
face. 


7.4.1 The querying process 


Attacks on the query resolution system itself are considered in, e.g., [72, 203, 204]. These 
attacks can be conducted by single peers or by a group of colluding peers. Specific tactics 
include: 


e misforwarding queries (or responses) and/or modifying them, e.g., forwarding away 
from the data object key, modifying the TTL, or modifying the forwarding record on 
each query (affecting the reverse-path forwarding mechanism); 


e generating a great quantity of phony queries to deplete the system’s available re- 
sources for query processing; 


e replying with stale or erroneous information when polled for forwarding table entries; 
and 


è simply discarding queries (or responses) rather than relaying them. 


Peers can assess their fellow peers not only by examination of the fate of their own 
queries but also by examination of all query and response traffic they receive. These as- 
sessments can be maintained in the form of "reputations" that can be pooled (for improved 
accuracy) by trusted peers and employed to detect malicious or faulty peers; see the next 
section. Peers with low reputations can be removed from forwarding tables and a "blacklist" 
of low-reputation (suspected) peers can be included, in a secure fashion, on each forwarded 
query thereby warning trusted peers that subsequently handle the query. In [235], a repu- 
tation overlay system is proposed to secure BGP. In [218], a method of "indirection" and 
authentication for secure routing is proposed wherein the identity of a peer is decoupled 
from its encryption key (signing identity), thereby facilitating periodic replacement of the 
encryption key. 

In [13, 125], security systems are proposed that can protect availability of content (data 
objects) in the face of denial-of-service attacks targeting individual peers, groups of peers, 
or areas of key space. Suitably redundant tables of data objects that can be cached among 
peers that are not proximal to try to thwart a single denial-of-service attack localized in 
space (as discussed earlier in the context of load balancing for flash crowds). 

Tor [60] is an onion routing system for anonymous Web surfing that employs both short- 
term and long-term encryption keys. Recall that Freenet, briefly described in Section 7.2.2 
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above, anonymizes file sharing within its community. Also, the privacy-preserving Tarzan 
system [77] uses, for example, additional "cover" traffic to mask certain activity. Such 
systems trade off anonymity with efficient use of resources. 


7.4.2 The downloading process 


With regard to the security of the downloading process itself, content pollution (ranging 
from spam to malware) is a potentially significant problem [221]. In the case of spam, it is 
possible that the data object may be easily perceived by the end user as not authentic. In 
the case of malware, a downloaded data object may be a trojan for an embedded virus; once 
the user accesses the data object, the malware may discreetly or overtly harm his or her 
computer and be spread further by the normal file-sharing operations of the P2P system or 
by other means. Instead of downloading the same data object from multiple peers, instances 
of a popular data object acting as a trojan for a virus can be potentially detected by applying 
a known hash function to the downloaded data object and comparing the result against 
that of the true, unmodified version of the data object; the legitimate hash values could be 
obtained from a secure website or trusted (super)peer. 

An interesting discussion of the potential for automated malware (worm) spread through 
P2P file-sharing systems is given in [223]. Such worms exploit vulnerabilities in the client 
applications of P2P systems. Collectively, the clients of a given P2P system may form 
a "monoculture" on the Internet with common vulnerabilities. The forwarding tables are 
built-in "hit lists" for the worm potentially enabling extremely rapid spread. Note that 
effective defenses to such worms need to detect and securely propagate (e.g., configure 
firewalls) faster than the worm itself. 

Some issues related to the security of cumulative reputation systems (in the context of 
promoting cooperation in uploading data objects) are discussed in Section 7.5.3. 


7.5 INCENTIVES FOR COOPERATION WHEN UPLOADING 


Systems to provide incentives for cooperation during the actual act of sharing (uploading) 
of data objects in P2P networks can be classified into three groups [7]: 


e monetary systems based on a global currency, 


e systems based on cumulative reputations (or cumulative tokens or credits) and asso- 
ciated referrals within trust groups, and 


e rule-based systems with memoryless "tit-for-tat" transactions. 


Using a global currency (e.g., a micropayments system [156, 185]), cooperative behavior 
of peers is rewarded with a form of payment that can be redeemed anywhere else in the P2P 
network, i.e., currency can even be used to make payment to peers with whom no previous 
contact has occurred. If significant amounts of currency are in play, an authentication 
mechanism may be required. Authentication can be implemented by a banking (certification 
and accounting) node or system. 
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Alternatively, the outcomes of transactions between a pair of peers, say A and B, can 
result in an adjustment of a tally of credits, tokens or reputations that A and B maintain 
with respect to each other. That is, if A successfully downloads a data object from B ina 
timely fashion, then A will reward B with credits or tokens or will increase B’s reputation. 
Thus, if in the not-too-distant future B requests a data object of A, then A will comply if 
B’s reputation is sufficiently high or if B can redeem enough tokens or credits previously 
accrued. Moreover, A may belong to a trust group of peers that (securely) pool their 
reputation information so that B can redeem credits accrued servicing A with another peer 
in A’s trust group; see Section 7.5.3. A superpeer in the trust group could maintain a single 
account of credits or tokens for each peer B outside the group so that any peer A within 
the trust group can (securely) deposit tokens into B’s account. Moreover, the manner in 
which credits accumulate over time could be such that the results of recent transactions are 
weighted more than those from the more distant past. Thus, this incentive system looks 
less tit-for-tat and more like a local currency system within the trust group. Such issues are 
discussed in greater detail in [93, 92, 217]. 

Incentive systems typically possess an "initialization" problem that impedes the first 
transaction involving a specific peer or trust group. Typically, each peer is given some 
initial measure of trust (a benefit of the doubt), for example, a peer is given a small initial 
token allotment [64] or is "optimistically unchoked" [23]. Such systems can be abused 
by so-called Sybil attacks [36] where a single peer assumes multiple identities in the P2P 
system in order to accumulate these small initial measures of trust for the purposes of 
free-riding, i.e., extracting significant benefit from the P2P system without contributing 
and cooperating in return. When peers unwittingly extend trust to Sybil attackers, or to 
peers colluding with physically different peers for the purposes of free-riding, the overall 
performance of the network can significantly degrade. A typical solution to Sybil attacks 
involves a centralized registration server that authenticates a unique identifier for each peer 
when it joins. Some of these issues are described further in Sections 7.5.3 (at the end) and 
7.5.4. 

In the remainder of this section, we briefly describe rule-based incentives for BitTorrent- 
like swarms and then give a detailed description of a specific cumulative reputation system. 
The section concludes with a discussion of game-theoretic frameworks that employ such 
incentives. 


7.5.1 Rule-based incentives of BitTorrent-like swarms 


Recall the brief discussion of search under BitTorrent at the end of Section 7.2.1. BitTorrent 
employs rule-based incentives wherein data objects are segmented into chunks (pieces) 
for the purposes of individual transactions that typically involve chunk swapping rather 
than single-chunk (i.e., client-server) transfers. For the purposes of distributing a given 
data object, a separate swarm of interested peers is formed. Peers that require significant 
numbers of chunks to complete the data object are called "leechers." Under a locally rarest 
first strategy, "seed" peers, possessing most or all of the chunks comprising the swarm’s data 
object, tend to first distribute the chunks that are rarest held (most demanded) by leechers. 

Since the peer arrival rate to a swarm may initially experience exponential growth [88, 
19], dealing with free riders is important (recall the initialization problem mentioned above 
in particular). In addition to efficient chunk swapping, BitTorrent employs a peer "choking" 
mechanism to deter free riders (briefly described in Section 7.5.4 at the end of this chapter). 
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Avalanche [11] proposes a network-coding approach to facilitate multicast dissemination 
of content that is segmented. Under Avalanche, each peer can distribute a single message 
that is an algebraic combination of all of the different chunks he or she has received so far, 
together with a specification of exactly how the chunks were combined. As with a FEC 
framework, the data object can be reconstituted upon receipt of a sufficiently large number 
of any such messages (whether unmodified chunks or messages formed by combining many 
chunks and other messages together) [76]. In a large-scale distributed system, the method 
of combining may be chosen at random to ensure, with minimal overhead, that a maximal 
number of received messages are linearly independent. 


7.5.2 A cumulative reputation system to observe the level of peer 
cooperation 


Consider a group of N peer end systems that subject one another to queries for data objects. 
A query (say from peer 7 to 7) together with a response (j’s response to i’s query) forms a 
transaction. For i # j, let Rij be the normalized reputation of end system j from the point 
of view of end system í, i.e., it will always be the case that 


Ss Ri; = 1 forall end systems 2. (7.8) 


had 


j, J#t 


As transactions occur, these reputation states will change. In particular, if i queries j and 
the subsequent response is that j gives 7 the requested data object (i.e., responds positively), 
then R;; will increase. In the following, we will study a specific kind of reputation system 
that estimates the propensity of end systems to cooperate (respond positively to queries). 

We begin with a set of definitions. Without consideration of reputation, let 7; > 0 be the 
fixed probability that end system j will respond positively to a query, i.e., j’s propensity to 
cooperate. In the following, a sequence of assumed-independent transactions is considered 
with R;; (n) representing the reputation of j from 2’s point of view after the nth transaction. 
We assume that transactions are independent so that the N (N — 1) vector R of reputation 
states Ri; will be a Markov chain on (Xy_1 \N, where © —1 is the N-dimensional sim- 
plex; see (7.8). The process R makes a transition upon the conclusion of each successful 
transaction. Let 


z 1 
Rin) = yo XO Rei(n) (1.9) 
k. ki 


be the mean reputation of ¿ after the nth transaction and let the response function 

G(T; 3 Ri) 
be the probability that j responds positively to ¿s query, where G is generally nondecreasing 
in both arguments. Also, G(z, R) = 0 only if R = 0. That is, end system j obtains and 


averages the reputations R;; from all other peer end systems k and modifies its intrinsic 
probability of responding positively, 7;, accordingly. 
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Now a specific mechanism for updating reputations as a result of transactions will be 


defined. If the nth transaction involves i querying j,then, with probability G(7;, Ri(n—1)), 


Ry(n-1)+C ifk=j7 4i, 
Riu(n) = 1+¢ (7.10) 
l R(n- 1) ifk £ j,i 
1+C ~~ 


for some fixed C > 0, i.e., Rj; becomes relatively larger only when the transaction ij 
succeeds. With probability 1 — G(r;, Ri(n — 1)): 


Rij(n) = Ry(n—-1) foralli Æ j, 


i.e., if the transaction fails (the data object is not transferred as requested), then there is no 
change in the reputation. Note that reputations of end systems may (relatively) decrease 
upon positive transactions that do not involve them. 

Let p;; be the probability that a transaction involves 7 querying j so that 


2 Dot = A; 


i j jti 
For n > 1, we can now directly derive 


E(Rij(n) | R(n — 1)) 


= |1- 5y Pik | Rij(n— 1) 


k, kei 
5 Ry(n-1)+C 2 
+ pis [Ry(n — 1)(1 — G3 (m5, Ri(n — 1))) + Fe Gils, Buln -1)] 
5 Rij(n-1 = 
F 5y pik [Rij (n — 1)(1 — Gr (ae, Ri(n — 1))) + Tyin DG (a, Raln —1))| 
k, kij 

1 ee pirGe (te, Ri(n — 1)) | Rij (n — 1) 

= = ik Gk (Tr, Ri(n — ij(n — 
1+C een j 

Spi Gy(r 5, R(n -1 7.11 

+ Ty Gh (Tj, Ri(n — 1). (7.11) 


In the first equality above, the nth transaction does not involve i querying in the first term, 
involves ¿ querying j # 7 in the second term, and involves i querying k Æ j in the third 
term. 

We will say that the response function G is e-bounded if it satisfies the following property: 
There is a strictly positive € < 1 such that, for all 0 < 7, R < 1, 


ex < G(r, R) < m. (7.12) 
Also, G is said to be separable if 
G(r, R) = 79(R) 


for nondecreasing g, where, by (7.12), g(0) > £ and g(1) < 1. For example, g(R) = 
e+ R0 — e). 
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Theorem 7.5.1. Ifthe response function G is separable and €-bounded, then 


jim E(u | RO) = =A forall iz j 
5 kki tK E 


Proof: For separable G, define 





P Pig ; ; 
Xij(n) = R,;(n) foralli 4 j andn > 0. 
0) Ral # 
By (7.11), 
C P , a , 
E(Xi;(n)|R(n-1)) = 1 — rai 5y pikG(Tk, Ri(n —1)) | Xij(n - 1), 


where (7.12) allows division by g(Rj(n — 1)) > 0. Since 


C y = Ce 
et PikG (Tg, Ri(n—1)) > — 5 PikTh =Z Q 
1+¢ k, kżi 1+¢ k, kži 
andl > œ > 0, 


E(\Xij(m)||R(n-1)) < (1-a) Xy(n -— 1). 


Now, define the random vector F,, = (R(0), R(1). ..., R(n)) for notational convenience. 
Note that 


E( 





Xij(n)| | Fn-2) = E(E(|Xi;(n)| | Fn-1) | Fn-2) 
(1 — a)? |Xij(n — 2)|. 


IA 


Repeating this standard conditioning operation iteratively gives 
E(|Xij(n)|| Fo) < (1 -a)”|X:;(0)| 


from which the theorem statement follows since X;;(0) is bounded and 1 > a > 0. oOo 


Interpreting Theorem 7.5.1, the reputation ER;; is a consistent estimator of "intrinsic" 
mean rate p;;7; of successful transactions ij. One can modify this reputation framework to 
observe more complex kinds of transaction outcomes. For example, the querying peer can 
estimate their download rate r (say in megabytes per second) and use r in (7.10) instead 
of the fixed constant C for "successful" transactions. Assuming that the transmission rate 
of a transaction is generally determined by the upload bandwidth (r) that the providing 
peer allocates to the file-sharing network’, this reputation system will estimate the relative 
values of quantity 7,7; for each peer j. 


'This assumption is supported by asymmetric residential broadband Internet access in which the downlink band- 
width is typically much larger than that of the uplink. 
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7.5.3 Modeling trust groups 


In this section, we discuss scalability and security problems facing deployed reputation 
systems. We also describe how to model a distributed, cumulative reputation system that 
could be implemented and secured by a referral system based on trust groups and light- 
weight message authentication and peer registration mechanisms; see, e.g., [31, 114, 115]. 

Potential attacks on cumulative distributed reputation systems include, but are not limited 
to, badmouthing and ballot-box stuffing, which are variations of Byzantine attacks [78], 
i.e., false reputation referrals and associated collusions. For the example of KaZaA, the 
reputation of each peer is stored in its own client. This fully decentralized approach can be 
subverted as seen with the KaZaA Lite client [121] which always reports a high reputation 
for itself. eBay reputations are centrally stored but are not therefore immune to attack via 
the methods described above [181, 182]. 

Similar to [31, 115, 149, 234], we can account for misrepresentation and subsampling 
of reputations by using the following instead of R; in the reputation model: 


= Lege Ajkih(Rje(n))Rki(n) 
Ay = sch a , (7.13) 


where the terms Ajg; can be used to represent how node k may misrepresent when polled by 
j for it’s reputation, i.e., Ajxi € [0, 1/Rpi(n)} and misrepresentation occurs when Aki E L 
The h-function parameters can be used to weight reputation information by the reputation 
of the pollee [149] (such reputation "weightings" can also be used in more general voting 
systems for distributed decision making). Examples are h(Rj,) = 1{R;k > 0j} and 
A( Ryn) = Rjik1{R;k > 0;}, where 0; > 0 is a reputation threshold that may be used by 
node j to define "trust". Trust and reputation are discussed in [57, 141]. 

Since the terms Aj4;h(Rjx(7)) > 0 depend on reputations only through R(n), we can 
generalize the model (7.11) to account for misrepresentation and subsampling of reputations 
by replacing in (7.11) the R; as defined in (7.9) with Rji as defined in (7.13), i.e., 





C 7 
E(Rij(n) | R(n-1)) = 1— To Aa —1))] Ryn — 1) 
BA enone 1 7.14 
T 14^” (Tj: Rjln — 1)). (7.14) 


As a special case, we can model federations of peers that are formed in part to pool 
reputation information in a more reliable manner, i.e., trust groups. That is, consider M 


groups {Nm }“_, of peers, where 


is the set of all N peers and |N,,,| > 2 for all m. This is modeled by taking 


Ay = { 1 if j,k € Nnm for some m, 


0 else. 


Note that {Nm}; need not be a partition, i.e., a single peer j could belong to more than 
one group (and note that we clearly need to assume that the denominator of (7.13) is always 
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strictly positive for all ¿, J}, n). Such local trust repositories can be securely implemented 
using one of a variety of existing light-weight authentication techniques within each group in 
a distributed or centralized fashion [25, 46, 179]. Obviously, the implicit assumption of any 
trust group 7 (assumed small enough to allow complete sharing of reputation information 
among its peers) is that, for all appraised nodes 2, Ajxi © 1 for all nodes j. k € Nan. Ina very 
large-scale P2P network, a deeper hierarchy of trust groups would form a complex "social" 
network. Instead of simply allocating an unknown peer a small reputation value (recall 
our response function Giz, R) > ex > 0 for all reputations R in (7.12)), a reputation 
for a querying peer could be determined by a routing operation through the social network 
(131, 222], i.e., one search process determines the providing peer for the querying peer and 
then another search process determines a reputation of the querying peer for the providing 
peer. 

Finally, recall Sybil attacks, which were briefly described for the context of the peer 
initialization problem at the start of Section 7.5. Again, a Sybil attacker employs many 
different identities and can try to create a phony "mutual admiration” society or attempt 
to harm the reputation of other peers through falsely negative referrals. Note that the 
reputation accrued by a Sybil attacker through cooperative action (uploading data objects) 
may be diluted among his or her multiple identities. A system of federated trust can limit 
the scope of harm of a Sybil attacker to within the trust group(s) to which he or she belongs, 
and specific strategies have been proposed to thwart Sybils within a group [58]. 


7.5.4 Discussion: Game-theoretic models of P2P systems 


Game-theoretic models of communication networks typically involve end users that behave 
in a rationally selfish manner. For P2P query resolution, recall the token-based system of 
GIA mentioned at the end of Section 7.2.3. Games modeling the sharing of data objects 
typically involve a mechanism that is designed to provide incentives for peers to cooperate, 
as discussed at the beginning of Section 7.5. 

Several authors have explored game models in which peers file share and adjust their 
reputation estimates of each other [68, 146, 147, 158, 159]. For a given transaction between 
two peers, the cooperation level of one peer will depend on the reputation of the other as 
described above. In the game, peers may modify their own "default" cooperation level 
(deciding, for example, whether to comply with a request, 7, and/or how much uplink 
bandwidth to assign to the transaction) in order to change their own reputation and thereby 
achieve a desired utility from the system (e.g., their target query success rate or mean data- 
object download time). Practical challenges to such game formulations include calibration 
of reputation levels (i.e., determining specifically what reputation levels correspond to a 
"cooperative" user [158]), and reconciling the time scales that are required to accurately 
estimate reputations, the time scales that are required to make decisions (transactions), and 
those of peer churn. Similar time-scale issues were behind the “quasi-stationary” ALOHA 
example of Section 4.6 wherein users act based on assumed accurate estimates of steady- 
state received quality of service @ for a given set of decision variables À and a fixed user 
population. 

Under rule-based BitTorrent, a peer may employ a "choking" algorithm to periodically 
refuse to upload to certain uncooperative peers, e.g., peers that are perceived to have allo- 
cated little uplink bandwidth in recent transactions. BitTorrent peers may “optimistically” 
unchoke certain peers, a policy that is especially beneficial to potentially cooperative peers 
that have recently joined a swarm (and have little to share) or to potentially rehabilitated 
free-riders that have increased their uplink (transmission) rates. 
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Problems 


7.1 Consider a simplified version of the query forwarding scenario of Section 7.2.4 
wherein there is a fixed population of peers, i.e., A = 6 = 0 so that N = f + w is 
constant. Let p = Ef /N be the fraction of peers that have cached the specific information 
that is the objective of the queries under consideration. Given f, simply suppose that before 
a query is resolved (i.e., received by a peer of the population f), it will be forwarded to 
N/f peers in the w = N — f population of peers that do not possess the information of 
interest. Using Little’s formula, show that 





7.2 Specify a Markovian model corresponding to the deterministic equations (7.2) and 
(7.3) for search by random walk with reverse-path forwarding. Carefully handle how the 
Markov chain behaves near f = 0 so that this state is not "absorbing." 


7.3 Consider n peers whose keys are chosen independently and uniformly at random on 
a unit circle and assign to each peer the arc segment for which its key is the left boundary 
point. Let A, be a random variable representing lengths of arc segments. 


(a) ProveEA, = 27/n œ~ o(A,), where ø is standard deviation and the approximation 
is close for large n. 
Hint: first argue that 





In —a2\"! 
P(A, >£) = ( a ) , 


So, the standard deviation of the segment size is comparable to its mean. 


(b) Now suppose that each peer has k keys assigned, where all nk keys are chosen and 
arc segments are assigned as for part (a). Let Bn,x be a random variable representing 
the total lengths of the k arc segments of a peer. Clearly, EBn, = KEAn, = 27/n = 
EAn, i.e., the mean total arc length assigned to a peer does not depend on k. Assuming 
that the lengths of the arc segments assigned to a peer are uncorrelated, prove 


a(An) 
Te 





o(Bn,k) 


Hint: Recall the law of large numbers from Section 1.11. Note: Chord suggests k = 
O(log n) [117], and one can use the conditional uniformity property (recall Problem 2.14) 
to show that B,,, is actually beta distributed with 0?(B, 4) ~ (n?k)~lasn — oc. 


7.4 Suppose |X| peers belong to a P2P system with addresses chosen in one-dimensional 
key space of size |X] independently uniformly at random. Also suppose that each peer is 
primary for a set of keys of fixed size 2r about the peer and that there are no secondary 
assignments. Find the probability that a region of key space of size k < |K] is not known 
to any of the peers, where k >> r. 
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7.5 Consider N > 1 peers (points) residing on a circle with unit radius. They are evenly 
spaced, i.e., the (shortest) arc length between nearest neighbors is € = 27/N. Each peer 
can forward messages to either of his nearest neighbors. Also, each peer x can forward 
messages to v > 1 long-distance peers L;(x), 1 < i < v, each of whose position on the 
circle is chosen independently and uniformly at random. Let D(m) = d(x(m),y) < 7 
be the shortest arc length (in any direction) between peer x(m) and data object y on the 
circle. Assuming that peers are aware of the position in the circle of a peer to which they 
want to send a query, a query from peer x(m) for y is forwarded to the peer x(m + 1) in its 
forwarding table that is closest to y, where peer x(0) originally launched the query for y. 


(a) Assuming for simplicity that D(m) >> € and that the distribution of the position of 
the long-distance peers on the circle is continuous (€ < 1), show that 


v+1 
E(D(m+1)|2(m)) = — (2- (1-2) (7.15) 





v+1 T 


D(m) ee +0 (e=) : 


(b) Repeat (a) for the case where D(m) approaches £. 


(c) Compare the search performance to Chord with similarly sized finger table at each 
node (especially as D(m) approaches £). 


Hint: Find 6(z) = P(miny<;<, d(L;(z),y) > z) and argue that the quantity of interest in 
(7.15) is — fae zd®(z)—D(m) Soim) d®(z). The second equality is Taylor’s theorem. 


7.6 With regard to the system and statement of Theorem 7.3.1, suppose that the key space 
is instead partitioned into equal-sized (rather than logarithmicly increasing) elements for 
the purposes of forwarding. How would the statement of Theorem 7.3.1 change? 


7.7 Consider the bitwise XOR distance of Equation (7.6). 


(a) Prove that it is a formal metric on {0,1}. Hint: To show the triangle inequality, 
argue that d(x, z) = d(x,y) @ d(y,z) anda+b>a@bforall a,b > 0 [152]. 


(b) Prove (7.7). 


(c) Explore the possibility of using the Hamming distance instead of an arithmetic (Eu- 
clidean) or XOR metric, i.e., d(x, y) is the number of different bit positions in which 
the binary representations of x and y differ. In particular, how do the "neighborhoods" 
in key space differ? 


7.8 Consider an overlay network of n peers for which there is a large number R of distinct 
routes, i.e., R = O(n?) > L, where L is the number of (one-hop) bidirectional "links" 
directly connecting a pair of nodes. Suppose that the links are unreliable, where each 
link l has a message-dropping probability b;. Assuming the message dropping events are 
independent, the route-dropping probabilities B, satisfy 


1-B, = [Ja -%) = Jana- b), 


ler lel 
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which implies 





Aw = log(1 B,) = 5 ar log(1 os bi) = 5 ar 1Ôl, 


ler ler 


where we are recalling the notation of Section 5.1. So, we can write this succinctly in vector 
form as 

A=A"S, 
where AT is an R x L matrix. Suppose that the link-dropping rates, represented by 4, are 
fixed but unknown. The problem formulated in [42] is to determine 6 from a small number 
k < L of "probe" routes for which the route-dropping probabilities are measured, Explain 


how this can be done. Hint: In one method, a k x L submatrix AT of AT that is of full 
row rank k is first found. 


7.9 Verify the claim at the end of Section 7.5.2 regarding the reputation system estimating 
Tiri for peers j, where r; is the uplink bandwidth (transmission rate) allocated by peer j 
for file sharing. 


7.10 Suppose the reputations are not normalized by 1 + C in (7.10), so that they simply 
accumulate the reward C for each successful transaction, and that the outcome of the 
transactions do not depend on the reputations of the querying peers, i.e., G(r, R} = m. Use 
the law of large numbers (Section 1.11) to show that 

Ne e OU Gera 

Bee: n aren PikTk 





APPENDIX A 


ADDITIONAL BACKGROUND ON 
ROUTING 


A.1 NETWORK GRAPH TERMINOLOGY 


Consider the network graph depicted in Figures A.1 and A.2. Note that it has unidirectional 
links (edges) that come in pairs, i.e., there is a link from node (vertex) v to node u if and only 
if there is a link from u to v as well. A node is responsible for and immediately cognizant of 
the costs of usage associated with all of its outbound links. Let £(v) be the set of outbound 
links of a node. Thus, £7! (l) is the single node for which / is an outbound link, i.e., the 
node transmits out on /. For example, in Figure A.1, 


L(f) = {6,8} and £°1(2)=b. 


Let M(v) be the set of nodes that are immediate neighbors (connected by a single 
outbound link) of node v, e.g., in Figures A.1 and A.2, the neighborhood of node a is 


N(a) = {b. f, d}. 
We can use this definition to define a neighborhood of a set S of nodes: 


N(S) = JN). 


ves 
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Figure A.2 A network graph with unidirectional links labeled by their cost ci. 


Note that by definition 
v € N(v), 
but it may be the case that 


SAN(S) # 4. 
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A unidirectional route (path) from node u to node v consists of a series of unidirectional 
links in the usual way. For the example in Figure A.1, the route labeled r = {1.2.4} 
connects node a to g. Alternatively, we can specify a route using nodes, e.g.,7 = {a.b. e.g}, 
where we note that the ordering of nodes matters in this notation e.g., route 


{3.8} = tay fd} Æ {f.a.d} = {6.9}. 


Given a measure of cost borne by a node/link to support a traffic flow on a route, one 
can define optimal (minimal cost) routes. To this end, let C(r) be the cost of a route r 
determined by the costs (or "weights" or "metrics") c; of its component links | € r. Note 
that (unidirectional) links connecting the same pair of nodes do not necessarily have the 
same associated costs. 

We will consider two general ways to determine C(r) from the c, l € r. The first way 
is when the costs are additive, i.e., 


Clr) = Ya. (A.1) 


ler 


The second way is when the route cost is the maximum of the associated link costs, i.e., 


Cir) = max cı. (A.2) 
Er 


In the following, the link costs are always assumed to be nonnegative, i.e., 


ca > 0 forall links /. 


E EXAMPLE A.1 


Recalling Jackson networks, if we associate a mean delay cost with each link, then the 
mean delay of a route is clearly the sum of those of its component links. Recalling 
loss networks, if we associate a loss (blocking) rate c; with each link and assume 
that loss events are independent and that that $`; € < 1, then the route cost is 
approximately additive, i.e., 


cir) = 1-[]a-«) 
ler 
x oe 


ler 


E EXAMPLE A.2 


In addition to or instead of delay considerations, a normalized energy cost associated 
with link / can be the basis of routing. For example, define 


TI 





Git = : 
be-i) 
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where b, is the energy (residual battery level) in node v which is available for trans- 
mission and 7; is the energy required to transmit a frame on link / [39]. In this case, 
we could identify route cost with that of the bottleneck link: 
C = : 
(r) max c; 
Note how an attribute of the node L7! (L) affects the cost associated with link 7. An 
optimal route in this context would have maximal bottleneck energy (or, equivalently, 


minimal cost C). That is, among a set of routes R connecting a pair of nodes, the 
routing algorithm would want to find 


minC(r) = minmaxcq (A.3) 
rER rER ler 
. be-i) 
= maxmin ; 





rEeR ler TI 


Other nonnegative link costs decreasing in bc-1q) and increasing in 7 could be 
used, including simply & = 1/b-1(,) or c; = 7, the latter resulting in an additive 
route cost. Hybrid delay and energy considerations are discussed in, e.g., [70, 108]. 
Finally, the assigned energy costs can be made more sensitive to current traffic load 
by substituting 7; for the current energy dissipation rate, i.e., the product of 7; and the 
frame transmission rate on the link. In [199], how such link costs lead to maximization 
of network lifetime is studied. 


Let 
a —>r ce 


be the subroute of r connecting node a to node c, where a and c are nodes that are along 
path r implicitly by this notation, i.e., a,c € r. Let a(r) be the origin (source) node of 
route r and w(r) be the destination (sink) node of route r. Thus, by the above notation, 


r = a(r) —, w(r). 


For example, in Figure A.2, the router = {b,a, f,d},a(r) = b,w(r) = d, and the subroute 
a —, f is just the link indexed / = 3. 


Lemma A.1.1. All subroutes of a minimum-cost route connecting two nodes in a network 
are themselves minimum-cost routes. 


Proof: Consider two nodes, a and f, along the (unidirectional) route r that is a minimum- 
cost route connecting node a(r) to node w(r). Suppose that a —>, f is not a minimum 
cost route connecting a to f, i.e., there is another route s connecting a to f such that 
C(s) < C(a —, f). Therefore, the route r’ created by connecting together routes 
a(r) >r a, s and f —, w(r) has lower cost than r, which contradicts r’s optimality. 
Consequently, the above assumption that a —, f is not a minimum-cost path is false. O 
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For example, in Figure A.2, route r = {b, a, f,d,e} (with C(r) = 7) is, by inspection, a 
minimum-cost route from b to e. Note that this route costs less than the direct (single-link) 
route from b to e (c2 = 10). Also note that the alternative (direct/single-link) subroute 
{a, d} from a to d has higher cost (cg = 3) than the optimal subroute {a, f.d} used by r 
(with C({a, f,d}) = 2). 

A loop is a route r that begins and ends at the same point, i.e., a(r) = w(r). A route is 
said to "possess a loop" if one of its subroutes is a loop. 


Lemma A.1.2. Among minimum-cost routes connecting two nodes in a network, there 
exists one that possesses no loops. 


Proof: By simply deleting a loop from a route r, we form a route r’ that connects a(r) to 
w(r) but with C(r’) < C(r), where equality holds only if the loop has zero cost. That is, 
removal of links from a route will reduce the cost for both (A.2) and (A.1). Oo 


For example, from Figure A.2, route {b, a, f, d,a, f, d,e} connecting b to e has a loop 
{a, f,d,a}. Removing the loop yields the loop-free route {b, a, f, d, e} also connecting b 
to e. For the remainder of this appendix, additive path costs (A.1) are assumed. Showing 
how the following results hold for path costs defined by (A.2) is left as an exercise. 


A.2 LINK STATE ALGORITHMS 


In link state routing approaches, nodes flood the network with the current costs of their out- 
bound links; see Section 7.2.2. The flooding could take place periodically and/or whenever 
an outbound link’s cost changes state. Suppose that each node v is currently in receipt of 
such link state information from every other node in the network graph. The nodes can 
therefore create a diagram of the entire network and label it with the (current) link costs, as 
in Figure A.2. Assume an at least temporarily static link state scenario in which each node 
has constructed the same diagram of the network. The objective of Dijkstra’s algorithm 
described in this section is that each node finds the optimal (minimum-cost) route to every 
other node in the network. Thus, for each packet, a node can make an optimal forwarding 
decision (first hop along the optimal path) to one of its neighbors. It then relies on Lemma 
A.1.1 to ensure that each neighbor will continue to forward the packet along the optimal 
route. 

The predecessor of node g on route r is denoted by z,(g). Because we assume that 
there is at most one (unidirectional) link from one given node to another, a special case of 
Lemma A.1.1 is: 


Corollary A.2.1. [fr is aminimum-cost (unidirectional) route connecting node u to v, then 
u —r Tr(v) is a minimum-cost route. 


With this corollary to Lemma A.1.1, we can state Dijkstra’s algorithm as it operates in every 
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node given current link-state information for the entire network. At node z, let S C V 
be the set of nodes to which a minimum-cost path has been determined, where V is the 
complete set of nodes in the network graph. The idea of Dijkstra’s algorithm is inspired 
by the previous corollary to build minimum-cost paths from the set S to neighboring nodes 
in V\S, i.e., nodes that have a predecessor node in S$ to which minimum-cost routes from 
x have been found. For each node y € S, Dijkstra’s algorithm maintains the predecessor 
node 7(y) along the minimum-cost route from x. Dijkstra’s algorithm is iterative and, so, 
we define v as the node most recently added to the set S. Finally, let «(z) be the cost of the 
best known path from x to z ¢ S so far and let c,_,, be the cost of the (single) link from v 
to z (i.e., implicitly, v and z are neighbors). 
Dijkstra’s algorithm operating at node z is: 


0. Initialize: S = {x}, v = x, k(x) = 0, K(z) = œ forall z Æ x, and z(z) = x forall 
zeV. 


1. Forall z € V(v)\S, if 


K(z) > kK(V)+ Cz, (A.4) 
then set: 


1.1. a(z) = v and 


1.2. K(z) = K(v) + Gp. 
2. For the node y € N(S)\S for which «(y) is minimal: 


2.1. v = y and 
2.2. S = S U {y}. 


3. If S = V, then stop, else go to Step 1. 


The initialization step simply states that the minimum-cost cost path from x to x has 
no links and zero cost. Step 1 determines whether better routes to destination nodes in 
N(v)\S are obtainable via the most recently added node to S, v, acting as a predecessor 
to a destination node. Note that a minimum-cost path to a new node y in step 2 may have 
been discovered by the current or by a previous iteration’s Step 1, i.e., we select among 
y € N(S)\S not necessarily y € N(v)\S. It is left as a simple exercise to show (by 
contradiction using Lemma A.1.1) that the path to node y selected in step 2 is optimal. 
Also, it is left as an exercise to modify Dijkstra’s algorithm for path costs are defined as the 
maximum of their component link costs. Dijkstra’s algorithm is described in greater detail 
in [3, 48}. 


E EXAMPLE A.3 


The following table shows the operation of Dijkstra’s algorithm at node a of Figure 
A.2. The states given are those found at the end of step 1 of each iteration. Iteration 
O is the initialization. With regard to iteration 1, note that the immediate neighbors 


of a are N (a) = {b,d, f}. Finally note that at iteration 6, S = V, the entire set of 
nodes. 
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Iteration S m,K(a@) m,k(b) am, K(d) n,xle) 7,K(f) 7.K(g) 
0 a a,0 oO ioe) x oc oO 
1 a.0 a,l a, 3 © a, 1 œ 
2 a,b a,0 a,l a,3 b, 11 a,l X 
3 a,b, f a, 0 a,l f2 b,11 a, 1 ee) 
4 a,b, f,d a,0 a,1 $22 d,6 a, 1 oC 
5 a,b, f,d.e a,0 a,l $52 d,6 a,1 €.8 
6 a,b, f,d,e,g a,0 a, 1 f.2 d,6 a,1 €,8 








By considering the predecessor nodes of optimal paths found by Dijkstra’s al- 
gorithm, one can construct a tree rooted at node a that spans the entire graph and 
summarizes the forwarding decisions from a. In this example, the optimal spanning 
tree rooted at a has two branches: a — f —d e — ganda —> b. Thus, all 
packets relayed or spawned by a that are destined to nodes f, d, e or g are forwarded 
by a to f. One can easily check that Dijkstra’s algorithm at f will result in a spanning 
tree that contains the branch f — d — e — g consistent with the optimal spanning 
tree rooted at a. 





As mentioned previously, a node could flood the network immediately after one of its 
(outbound) links changes state, where individual floods may be made economical with the 
use of flood identifiers, TTL mechanisms (see Section 7.2.2) and a kind of reverse-path 
forwarding [139]. But to further economize on routing overhead and to make routes less 
sensitive to, say, faulty links whose costs may change frequently, a node could be permitted 
to only periodically flood the network with its link state information. The resulting staleness 
in link state information (even for small periods of time) may, however, lead to the temporary 
use of suboptimal routes or attempted use of no-longer existing routes. 


A.3 DISTANCE VECTOR ALGORITHMS 


Distributed Bellman-Ford algorithms involve exchanges of routing tables only among 
neighbors instead of link state information flooding the network. The routing table of 
a node x would have an entry for each destination node z # x consisting of its best route 
r to z and the cost C (r) of that route. Suppose z is in receipt of the current routing tables 
of all of its neighbors. For a given destination node z, node x considers the routes to z via 
each of its neighbors y € N(x), i.e., the first hop y € N(x) after the origin z is considered 
rather than the predecessor 7(z) to the destination z. 

Assuming additive link costs, let C(y — z) be the current minimum cost among routes 
from y to z as advertised by y to x. Under the Bellman-Ford algorithm, node x computes 


min ¢zo4,+Cl(y >z 
ING Y (y ) 
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Figure A.3 A network graph with unidirectional links labeled by their cost cz. 


for each destination node z. The route that achieves the minimum becomes the entry 
for destination z in x’s own routing table. The route minimum costs thus computed are 
advertised to x’s neighbors in turn. 


E EXAMPLE A.4 


The action of the distributed Bellman-Ford algorithm for the three-node network 
of Figure A.3 is given in Figure A.4. The entries in the distance vectors that are 
exchanged are indicated by circles. For example, after the initial step 1, node a 
transmits this distance (route cost) vector to each of its neighbors: 


Destination Route Cost 


b 2 
d 5 





After step 2, there are no further changes to the routing tables of the nodes for the 
static network of Figure A.3. 


The following result can also be argued by contradiction and its proof is left as an 
exercise. 


Theorem A.3.1. For a given, fixed set of link costs, the distributed Bellman-Ford algorithm 
will not terminate before optimal routes have been determined by all of the network’s nodes. 


A distributed Bellman-Ford algorithm can be designed in which only the next-hop- 
from-the-origin node is kept in the routing tables rather than the complete route. In this 
case, the entries in the exchanged routing tables are simply pairs of destination node and 
route cost, i.e., distance vectors. Link costs may be very sensitive to traffic load, i.e., 
sensitive to the routing decisions themselves. Links may even be severed (their costs become 
infinite) by excessive traffic load, hardware or software faults, and, especially in a wireless 
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Figure A.4 Action of link-state algorithm. 


setting, environmental conditions or node mobility. In response to changing link costs, 
the use of distributed distance vector routing may result in slow convergence to optimal 
routes (especially when link costs increase) because routes with loops are not immediately 
excluded from consideration. Simple techniques exist to address these problems [139, 172]. 
For example, if x itself is y’s next hop to destination z, then y advertises infinite cost to 
z when it sends its distance vector table to x. Thus, x will not use y as its next hop to z, 
avoiding a possible two-hop loop back to x. This technique is called poison reverse. Other 
techniques such as DSDV [171] and hold-down (delaying updates; see p. 358 of [172]) 
also address the problem of route loops. BGP avoids loops by carrying (entire AS) path 
vectors rather than just distance vectors [101, 180]. 
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APPENDIX B 


SOLUTIONS OR REFERENCES FOR 
SELECTED PROBLEMS 





Chapter 1 


1.12 Note that if either one of the nonnegative quantities E(X 1{X > 0}) or E(~X1{X < 
0}) is finite, we can define 


EX = E(X1{X >0}) -E(-X1{X <0}. 


1.13 The values of a and b that minimize the MSE are 





b = E(X —aY) = EX —aEY, 
ae cov( X, Y) 
o var(¥) ` 


1.15 The MGF of a binomial random variable is 


n 


Maal) = N (ea — q)” -* 


k=0 
= (ge®°+1-q)” 


la + qle? = D) 


ngle? -1) 
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Therefore, as q — 0 and n — œ in sucha way that ng —> À, 
Mnal) — exp(A(e? — 1)), 
which is the Poisson MGF as desired. 


1.16 (a) Y, is Poisson distributed with à = EY. 


© 2\n 
EY; = ye 





Thus, var(¥,) = EY? — (EY)? = À. 


(b) Let my be the MGF of Z) = (Yy — )/ VX. Since the MGF of Y; is exp(A(e® — 
1)), 


mal) = exp(—VA0 + àe? — A). 


Expanding about 0 = 0, we get 


2 
oA Ui ee 1 
e 1+ ex 5 +0(5 


where we recall that o(x) ("little oh x") is a function satisfying o(x)/x — 0 as 
x — 0. After substituting this expansion, we get 


m)(6) = exp ($ +A+0 (5)) 


6? 
= ex (5) as À > œ, 


which is the standard Gaussian MGF. 
1.19 Recall that a CDF is a nondecreasing mapping F : R — (0, 1}. Thus, 
P(F-'(U) <a) = P(U<F(2)) = F(z) 
where the second equality is simply due to the fact that U is uniform on [0, 1). 


1.20 The probability that all have a unique birthdays is 


ey 


1 2 n-1 
365"/nl ~ 1(1- 35) (1- gg) 0- a) ey 
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where the quantity in the denominator of (B.1) is that obtained by choosing with 
replacement. So the answer is obtained by setting 1 minus the quantity in (B.1) to 
0.5 and solving for n. 





1.21 The likelihood that one spoofed response fails to match any of the transaction IDs of 
all of the forwarded queries is x = (1 — 2~1®)%. So, the likelihood that all spoofed 
responses fail is x°. So, solving 

(1 — gree = (1 T g-16)n? = 0.5 
gives n = \/log(0.5)/log(1 — 2716) rounded up to the nearest integer. See [208]. 
Chapter 2 
2.3 Consider a differentiable function ® on R* satisfying 
(0) = 1, (B.2) 
lim (z) = 0, (B.3) 
LIM 
O(r+y) = (x)O(y) forall x,y € RH. (B.4) 
So, we can interpret ® = 1 — F, where F is the CDF of a continuous, nonnegative 
random variable with the memoryless property. We are required to show that, for 
reERt, 
(z) = exp(r’(0)), 
where ®'(0) < 0. We can do this by differentiating (B.4) with respect to y, evaluating 
the result at y = 0, and then integrating with respect to x. That is, beginning with 
(B.4) above, 
(+y) = O(x)O'(y) 
=> (x) = (r)6'(0). 
Therefore, by integrating, we get 
(x) = Aexp(r6'(0)) 
for some constant A. Finally, (B.2) above implies that A = 1 and (B.3) implies that 
®'(0) < 0 as desired. 
2.5 


P(X, + Xə < £) 


€ € 
f / dye7 2 Noe *2”? dro dr 
0 zı 


f \ye7™™2 (eons — ee) dzı 
0 








a Ay — (A1 +à2)E — Àz£ -AlE 
= ees a )-e (1-e ) 
M1 
= e+o(e))(1 — Age + of€ 
Trae ag (On Fade + o(e))(1 — Aze + 0f2)) 


x (Are + ofe)) 
= Ave — Ave + Of) 
= ofé). 
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2.6 Find the only n x n matrix R such that 72 R = a? for all initial distributions zo. 
2.14 Hint: 
P(X; € (zi, zti + dz;), i=1,....n, NU) =n) 
= Ne dri- (i he Nee) ss) e7MT-2n) 
i=2 
and N (T) is Poisson distributed with parameter AT. 


2.16 Using the hint and substituting p = |J|/{I| give 


ee (AÊ) eTA 


r=0 


Eexp(sN* (1)) 





r! 


=, A(\L\e* a tR 


= apale =A) 
as desired. 


2.22 See [130]. 
2.23 Po = 2, Po, = 3, Pia = 1, Poo = 1, otherwise P,,,m = 0. 





Chapter 3 
3.6 
QU 
P(W(t)<z) = P j ST 
j=l 
= SP 5 <z ]|P(QE=n) 
n=0 j=l 
Me eek T pry? te Hy 
e pa n! 0 n! dy 


3.8 (c) Let W be distributed as the steady-state workload and X be an exponentially 
distributed random variable independent of W. By PASTA, upon arrival of a typical 
batch, the workload distribution is that of W. But, the second job of the typical batch 
sees a distribution of W + X. Thus, the distribution of the workload upon arrival of 
a typical job is that of W + X/2. 


3.9 (b) 3À < pu. 
(c) P(Q>0)=1-—p=1-3A/p. 


(d) q1,0 = u > 0 but go.1 = 0. So, the detailed balance equation cogo,ı = 7141,0 
is not true and, therefore, the Markov chain is not time reversible. 
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(f) O= ae On4n.3> lêz 


0 90,300 — (93.4 + 93.2)03 + G4.304 


= og — (At 2u)o3 + 204. 


li 


(g) Considering the likelihood that an arriving job is blocked and using PASTA to 
condition on the queue backlog upon arrival of a batch, 


1 2 
302 + 303 + 04. 


3.10 Consider a real time t > @ and let i = A[@, t]. For an arbitrary single-server queue, 
W(t) = (W(a)-(t-a))*, 


where a < tis the most recent (largest) job arrival time prior to time f; see Figure 
3.5. In this case 
i— 1 
À 


By taking expectations of both sides of the previous equation for W, we get 


a = 0+ 





EW(t) = E(W(a)-— (t-a). 


To conclude, recall that t — a is uniformly distributed on [0, 1/A] (since f is a "typical" 
time). 
3.14 (a) By PASTA, the blocking probability is 
K 
fe Soe ee, 
La pape +p" 
see Section 2.5.1. 


(b) The net arrival rate is (1 — b)A jobs per second. So, by Little’s theorem, the 
sojourn time of jobs not blocked is 





K g 
(1 = b)A x 2k! oa 
r= P" 


3.15 See Examples 5-12 and 5-13 of [226]. 


Chapter 4 


4.1 See [127]. 
4.3 (a) For the packet of length / arriving at time T, compute its delay 


ô = max{d T} +$- T. 


If 6 > o/p then discard the packet, else set 


d = +a. 
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Thus, only a single state variable, d, is required. 
4.8 See [73, 227]. 
4.10 See [109]. 
4.11 See [109, 110]. 


4.12 Assuming the nodes are ordered, the number of conflict-free assignments is 
K(K —1)(K —2)---(K —N +1). Thus, the probability of collision in a time slot 
is 
K!/N! 
NK `’ 





4.13. (a) The vector s has entries s; = yj Rjo* /(Who,;) and the matrix A has entries 
Ajj = 0 and A; k = ¥;Rjho,n/(Whs,;) for j #k. 


(b) If det(A) # 1, then the unique fixed point for the unconstrained dynamics 
exists: 


ph = (I-A), 


(c) If the eigenvalues of A all have modulus < 1 (equivalently, A is doubly sub- 
stochastic), the unconstrained iteration p will converge to p". 


See [228]. 


Chapter 5 


5.3 A detailed balance equation when zn > 0 is 
o(£) UnTn,m = O(n mZ) Um"m,n- 
After substitution of the product form expression for o and cancellation, we get 
Antam = AmTmon- 


This is clearly not true in general, so the Jackson network is not time reversible in 
general. 


5.7 Following the hint, 


Br 
E (F1m > 0) 


I 


E(E (Els) [Ne >0) 


E (zE: INe) |N: > 0) ; 


So, by the conditional uniformity property of a Poisson process, 


EBL) = Me; f HX() € Blas 
0 
= N,P(X €B). 
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where the last equality is due to the stationarity of X. Substituting this into the 
previous equation gives 


B; 
E pas 
( Nt 


The desired result then follows from PASTA. See the appendix of [150]. 





Ni > 0) =P(X € B). 


5.8 (a) At any given time, the average number of mobiles in the cell is 
L := é6n(Q/2)°. 


The average sojourn time of a mobile in a cell is 


Ymax f7 A sind dé 
W = i: I ACR bida 
Vmin 0 v T 
2A. ma 
= — = f(v)dv, 
T v 


Vmin 


where 2A/7 is the average length of a chord in a circle of diameter A. Thus, 
by Little’s formula, 


Vmax 


where 8 := | ae + f(v)dv is assumed finite. Clearly, the calculation also 
works if velocity and direction of a mobile in a cell are dependent. 


(b) In this case, simply replace the term f(v)/z in the first expression for W above 
by the joint distribution f(6, v). 


Chapter 6 


6.5 See [124]. 


6.6 (b) U}(xž) = M* + Ul(a*)a*/c > M* and, since U/ is decreasing (U, is 
concave), this implies (U/)~!(M*) > 2%. Summing over r gives V(M*) > 
c = V(M), where the function V = > .(UL)~1. Finally, note that V is a 
decreasing function. 





Chapter 7 
7.1 Since each query results in an average of E(.NV/ f) new cache entries in the peer group 
N, 
1, N q N q 
Ef = Np = —-(qẸE—) > +Ż. = —, 
ji f n Ef pp 


where the second equality is Little’s theorem and the inequality is Jensen’s. 
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7.3 (a) Itis straightforward to check that 


Qn 
2 
EA, = ii P(A, >a)dr = Z. 
0 n 


To compute the standard deviation ø (An), compute the PDF — -$ P(A, > £), 
directly evaluate EA? (integrate by parts twice), and substitute into o?( An) = 
EA? — (EA,)°. 


(b) Since the variance of the sum of uncorrelated random variables is the sum of 
their variances, 


a? (Bn k) = ko? (Ank). 


7.5 Note that 





sore A) 


7.8 Consider a k x L submatrix AT of AT with rank k = rank(A) < L and the 
corresponding subvector A whose entries are all estimated, i.e., 


and follow the hint. 


A = A%6. 
An estimate of the link dropping rates can then be computed via a generalized inverse 
of AT giving the minimum-norm solution to the underdetermined set of equations 
above [30]: 


6 = A(ATA)“!A. 
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